Friday, April 26, 2013

Dell PowerEdge 2950: Silence the Noise, Part 2

First, a thank you to Neil for spurring my renewed energy into finding a solution to the 2950's noise level.  For the background of this project, check out Silence the Noise, Part 1.  In this post, I relate my research findings from my search for suitable fan replacements.



The Fans

To find a suitable replacement, we must first analyze the existing fans so we know the baseline for comparison.  Dell PowerEdge 2950's come with Delta brushless, axial fans in a 60mm x 60mm x 38mm form factor.  Dell appeared to utilize three variants, with Dell-approved replacement part numbers including JC972, PR272, YW880, and DC471.  The ones in my 2950 are JC972, for which the corresponding Delta model number is PFC0612DE.  It should be noted that some sites report the thickness as 35mm instead of 38mm, but according to the spec sheet, the proper thickness to keep in mind is 38mm.

Here's a recap and quick reference for the 2950 fans:

  • Dell Part Numbers: JC972, PR272, YW880, DC471 
  • Delta Part Number: PFC0612DE (I'm sure there are others)
  • Form Factor: 60mm x 60mm x 38mm
  • Air Flow: up to 67.8 CFM
  • RPM: up to 12,000
  • NOISE: 61.5 dB (one fan!!)
  • Voltage: 12V
  • Termination: 4 wire
  • Features: PWM Control
The key elements we need to keep in mind for the replacement are the size (60mm x 60mm), air flow, noise, termination, and features.  

To see like-manufacturer replacements, I checked Delta's website. Delta has a list of currently available fans in a similar form factor if you put in the correct search parameters.  However, in the comments section of the hacking the BMI post, other PowerEdge 2950 owners reported that they swapped the 38mm thickness for thinner fans and they worked fine as long as the replacements had PWM control and a 4-wire termination.  Here are some photos for reference:

Fan Label
4-Pin Connector

Now, if you've read my previous post, you know that I removed two fans leaving me a max of 120 CFM (~63 per fan).  During that time, my CPU temp did not increase above 25 degrees.  That means that across all four replacement candidates I can lower the CFM to 30 CFM max for each fan.  Your mileage may vary.


On arnuschky's blog, in the comments someone reportedly fitted 60mm x 60mm x 25mm fans, so we don't have to stay with the 38mm thickness.  A forum post at overclock.net provides insight into re-mapping the power pins from different fans into Dell's power connectors.

Keeping that mind, here is a list of potential replacements:

As you can see by visiting the links to these fans, all the power connectors are different with maybe the exception of the last one and would need to be re-pinned.  Also, note that they're all thinner, coming in at a 25mm thickness instead of 38mm.  Instead of trying to re-pin the power connector, I decided to check Delta's list of model numbers to see if they had a fan that I could use to more easily replace the JC972.  To be posted in part 3...


24 comments :

  1. I'm dealing with this right now. Hope you can find something. I've got 4 of these sitting on top of each other in a server closet about 6 feet from my desk. I've been toying with just rigging up some 80 on the inside of the case as I haven't really found a suitable replacement fan yet.

    ReplyDelete
  2. Wow, I just got around to taking the fans out of my 2950 to inspect them and I see there's now a part 2. I need to get going! Question about the Dell 7K412 fans for the PE 2650. I've looked at images of a few of those and they appear to have 3 wires, but using the 4 pin connector. Think those would still work? I'm tempted to try them out to keep from having to re-wire. I'll let you know if I do.

    ReplyDelete
  3. Hi Neil, great question! I'm still trying to determine the pinout configuration myself. This site (http://www.aliexpress.com/item/Delta-PFC0612DE-60mm-DC12V-1-68A-Strong-Cooling-DC-Fan-4P-with-PWM-Supportive/570585518.html) has the following in its description:

    "Lead Wire:
    Red Wire Positive (+)
    Black Wire Negative (-)
    Yellow Wire Speed Test
    Blue Wire PWM Control"

    This mapping follows the diagram at this forum post (http://www.setiusa.us/showthread.php?2480-PWM-Fan-Pinouts-Industry-Standard-amp-Dell-configurations).

    It looks like we need all four wires. I noticed Delta model number AFB0612EHE hsa 4-wire support and the pre-mapped pinout, but it's only slightly less noisy (http://www.aliexpress.com/item/Delta-6038-6cm6-cm-violent-fan-12V-1-68A-four-wire-support-pwm-AFB0612EHE/625696386.html).

    I just stumbled across this fan (http://www.aliexpress.com/item/DELTADelta6025-Cooling-fan-AFB0612EH-Four-wire-fan-Cooling-Fan/511541379.html). It has all four wires, bit with the wrong motherboard connector. It has a ~38CFM and a max ~48dB. I'll see how easy it is to pull out the wires of the JC972 and post an update once I do.

    ReplyDelete
  4. Oh my goodness, Matthew... Your a god send. Please post "Part 3" as soon as you can. I am in dire (WAF) need to silence this beast.

    ReplyDelete
  5. I have consolidated my VM onto a new server and plan to turn this 29050 off until I get a solution. Further more I have 2 exact same models of the 2950 and one runs at 7500RPM and the other at 9200RPM. I am shutting down the 9200RPM.

    ReplyDelete
  6. I'm not sure if you have done it already but try updating your BMC firmware. After I updated my Baseboard Management Controller firmware I noticed the fan speed slopes had changed. The fans now have a lower minimum speed and are noticeably quieter.

    Here is a link to the Dell Poweredge 2950 BMC firmware update v.2.50, A00

    https://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-2950?driverId=4NNNG&osCode=WNET&fileId=3078114158&languageCode=en&categoryId=ES

    ReplyDelete
  7. So I've gone all out trying to quiet down my Dell PowerEdge 2950 Gen II with 2x Xeon x5355 since my last post. I have to say, it was not a straight forward process.

    To start, I followed your advice and picked up some Top Motor fans as replacements. I picked the Top Motor FA60TM3BMP as my replacement fan. It turns out that the Top Motor fan you suggested (FA60TM3BEP) isn't being carried anymore at BestByte.com and they just haven't taken the listing down. A representative at BestByte named Barry was very helpful in pointing me to the Spec sheet for top motor fans they did carry. (http://www.dynatron-corp.com/en/product_detail_1.aspx?cv=20-72-112&id=53&in=0)

    The Top Motors seem to be just a re-brand of Dynatron parts. According to the spec sheet the Top Motor FA60TM3BMP corresponds to the Dynatron DF126025_M. With 7000RPM, 38.28cfm, 48.1 DBA, and 13.55 mm/H2O of static pressure, these fans should give me an acceptable margin for error.

    Now for the frustrating parts. All the replacement fans needed to have their connectors swapped with the stock fans, as well as having the closed mounting holes opened up. I tried a simple repin but the pins are not compatible for a simple swap. Specifically, the Top Motor pins were too large for the stock fan connector. My solution was to simply cut the wires, swap the connector leads, solder the wires back together, apply some heat shrink tubing and voila. Some Top Motor fans with proper connectors.

    I also discovered that the power supplies had the same high speed fans as the stock cpu coolers. I solder swapped the connectors for those as well with two more Top Motors but found out that the server won't get past the first POST screen when the power supply fans are at too low of an RPM. For the time being I simple reinstalled the stock PSU fans until I learn more about BMC hacking. A good Hacking guide can be found here. (http://projects.nuschkys.net/2011/11/15/how-to-adjust-the-fan-thresholds-of-a-dell-poweredge/)

    Some users on the Hacking guide thread have commented that the BMC hacking guide works on the 2900 as well as the 2800. I'm going to go out on a limb and assume it also works for the 2950.

    One more thing. Unrelated but someone may find it useful. I was able to replace a 2950 Gen I motherboard for a 2950 Gen II motherboard and everything worked just fine. No weird boot up issues or anything. With the Gen II motherboard I now have quad core cpu capability, and I was able to install those nice quad cores without issue.

    ReplyDelete
  8. Some pictures to clarify what I did.

    Staggered wire cuts for easy heat shrink tubing.

    (http://i77.photobucket.com/albums/j61/PockyFreak/staggeredwiresoldering.jpg)

    Heat shrink tubing applied

    (http://i77.photobucket.com/albums/j61/PockyFreak/heatshrinkapplied.jpg)

    I used a hacksaw, long handled pliers, and a Dremel cutting wheel to open up the closed mounting holes.

    (http://i77.photobucket.com/albums/j61/PockyFreak/closedtoopenmountingpoints.jpg)

    modified fans in cages.

    (http://i77.photobucket.com/albums/j61/PockyFreak/modifiedfansincages.jpg)

    modified fans in Poweredge 2950

    (http://i77.photobucket.com/albums/j61/PockyFreak/PE2950modifiedfansinstalled.jpg)

    Full system view with Ati 5450, 4 port USB3 pcie card, and three port firewire pcie card installed. Modified extra length molex cable was required for supplemental power for expansion cards.

    (http://i77.photobucket.com/albums/j61/PockyFreak/fansinstelledinPE2950.jpg)

    System at idle with 21c ambient

    (http://i77.photobucket.com/albums/j61/PockyFreak/Poweredge2950atidlewithmodifiedfans.png)

    system at full load with 21c ambient

    http://i77.photobucket.com/albums/j61/PockyFreak/Poweredge2950atfullloadwithmodifiedfans.png

    ReplyDelete
  9. Hi Kevin,

    Nice work! You really ran with it! I appreciate you posting your experience! Have you used a sound meter to measure the decibels? I'm interested to know.

    I've been so busy the past few months finishing some other projects and competing in cyber challenges I haven't had the time to tackle mine. I'm hoping to do so in the next month, following in your first footstep by upgrading the BMI. I did find some other fans that have potential, but I have to revisit my research. I don't know how to solder, and although I've heard it's simple, my goal is a complete plug/play swap.

    Did you have fun taking off the fan casing? I tried it myself and almost broke the dang thing before realizing the pressure releases!

    Lastly, would you mind if I recounted your efforts in a "guest" blog post? Aside from those few links on my "Part 1", there aren't many other comprehensive guides/steps out there.

    Regards,

    Matthew

    ReplyDelete
    Replies
    1. By all means. I'd be happy if you shared my experience on your blog. I actually just finished altering my BMC critical fan threshold settings and had some more info to share. I'm low on time now but I'll post more soon.

      Delete
  10. I can understand your sentiment about wanting completely plug and play components. I just don't think you will find dell compatible parts that will have a low enough RPM for your needs.

    If soldering is out of the question you still have a few other option available to you. All of the wires on the fans are 22 AWG width so you could probably use simple automotive wire butt connectors.

    Some examples of Butt connectors.

    (http://www.ebay.com/itm/50-Car-Audio-Wire-Butt-Connectors-Red-NYLON-22-18-Ga-AWG-Gauge-Terminals-Crimp-/390448885773?pt=US_Car_Audio_Video_Connectors_Terminals&hash=item5ae891300d&vxp=mtr)

    crimped butt connector
    (http://www.google.ca/imgres?start=147&sa=X&hl=en&biw=1707&bih=920&tbm=isch&tbnid=Ip86MIiCQ6TSiM:&imgrefurl=http://www.quickcable.com/products.php%3FpageId%3D386&docid=rl8PoEbuGGP-cM&imgurl=http://www.quickcable.com/images/display_images/d_connectors_solderless_pvc_parallel-butt-connector_3.png&w=750&h=500&ei=V7IPUq_eNNDyyAGqxYAw&zoom=1&ved=1t:3588,r:70,s:100,i:214&iact=rc&page=5&tbnh=181&tbnw=256&ndsp=42&tx=141&ty=78)

    A lighter would work on these
    (http://www.ebay.com/itm/10-Heat-Shrink-Solder-Butt-Splice-Terminal-Connectors-Auto-Electrical-1-5-6-0mm-/350832624871?pt=LH_DefaultDomain_2&var=&hash=item51af4104e7)

    (http://www.google.ca/imgres?sa=X&hl=en&biw=1707&bih=920&tbm=isch&tbnid=_IU4-Sz8mgVzhM:&imgrefurl=http://www.tech-cor.com/AutoResBulletin/1995-4/1995-4.htm&docid=sQ6060tSDpNBJM&imgurl=http://www.tech-cor.com/AutoResBulletin/1995-4/images/1995-4-IMG01.JPG&w=380&h=248&ei=BLMPUvnzGLKEygH3t4E4&zoom=1&ved=1t:3588,r:28,s:0,i:171&iact=rc&page=1&tbnh=181&tbnw=277&start=0&ndsp=30&tx=139&ty=80)

    crimp tool
    (http://www.ebay.com/itm/100Pcs-Nylon-Wire-Butt-Connectors-Terminals-Crimp-Insulation-/360517137064?pt=LH_DefaultDomain_0&hash=item53f07ef2a8)

    you could use simple pliers instead of a crimp tool. You just have to be careful not to over pinch the connector.

    All things said and done though, soldering would really give you the best and most compact connections.

    ReplyDelete
  11. I repair and refurbish quite a bit of consumer computer equipment so The pressure releases on the cages didn't take too long to figure out.

    Now to get the BMC firmware properly altered. That was a whole other level of frustration. Let me walk you through just how ridicules the process was for me.

    I started out by following Arnuschky great guide for getting a live Ubuntu distribution up and running on a thumb drive. Things started to go south quickly though. Apparently Ubuntu has removed several of the commands you need to run Arnuschky BMC editing script in version 12.04.

    Link to guide
    (http://projects.nuschkys.net/2012/04/06/how-to-get-ubuntu-live-running/)

    I started out by getting The BMC monitoring software and latest version of python with the following Terminal command.

    sudo apt-get install freeipmi freeipmi-tools openipmi ipmitool python wget

    This worked out fine and dandy. On to the next step in the guide.
    (http://projects.nuschkys.net/2011/11/15/how-to-adjust-the-fan-thresholds-of-a-dell-poweredge/)

    Query the internal sensors with this Terminal command.

    sudo impi-sensors

    Still looking good. All system go.

    Browse to your model server and download your latest BMC firmware.
    (http://support.dell.com/support/downloads/)

    Used the following Terminal commands and was greeted with errors

    sed -i 's/#!\/bin\/sh/#!\/bin\/bash/' BMC_FRMW_LX_R223079.BIN

    sudo chmod 755 BMC_FRMW_LX_R223079.BIN

    sudo mkdir bmc_firmware

    sudo ./BMC_FRMW_LX_R223079.BIN --extract bmc_firmware

    The "--extract" command apparently is not included in Ubuntu anymore. After much digging I was able to figure out how to make the command work again. You had to enter the following.

    apt-get rpm

    After that, rerun that last command

    sudo ./BMC_FRMW_LX_R223079.BIN --extract bmc_firmware

    Getting closer now. On to the next step. Time to get the patcher and then patch that file.

    sudo ./BMC_FRMW_LX_R223079.BIN --extract bmc_firmware

    cd bmc_firmware

    wget https://raw.github.com/arnuschky/dell-bmc-firmware/master/adjust-fan-thresholds/dell-adjust-fan-thresholds.py

    sudo chmod 755 dell-adjust-fan-thresholds.py

    sudo ./dell-adjust-fan-thresholds.py payload/bmcflsh.dat

    All things looking great. I selected option 3 for the 2950 and edited my fan thresholds to 14 which gave me an RPM lower limit of 1050. I then tried to write the setting to firmware but got an error. The error said "./disneyinstall.sh: 35: typeset: not found" This turns out to be a bug where the hapi/opt folder was not extracted. That folder is where the firmware writer is located. The following commands fixed the issue.

    cd hapi

    sudo tar zxvf hapi.tgz

    cd ..

    Now try patching the firmware again.

    sudo ./dell-adjust-fan-thresholds.py payload/bmcflsh.dat

    And that should be it. Fans thresholds adjusted and no more idle to max fan speed cycling.

    On a side note, just for curiosities sake I tried the procedure again but in Fedora 19 32bit and had none of these little hiccups or errors. If I had to do it again I would advise using Fedora instead of Ubuntu. Just remember, Fedora uses the "yum" command instead of "apt-get"

    ReplyDelete
  12. This comment has been removed by the author.

    ReplyDelete
  13. After all that you think things would get easier, right? Replacing the power supply fans turned out to be an exercise in futility. First off, I found out the hard way that the wire colors are not industry standard. They aren't even Dell's standard either! They are Dell's other standard apparently.

    I'll illustrate.

    (http://i77.photobucket.com/albums/j61/PockyFreak/fanpinout.png)

    After I a bunch of soldering and de-soldering I finally was able to make the PE2950 boot up past the first screen of POST. Only problem is that now the system rebooted right in the middle of Loading Windows. I don't have a definite answer but from what I can gather, The BMC can't read the PSU fan speeds when they get below about 2000RPM. And since the fans start out at full speed when the system boots up and slowly lowers the RPMs to an idle of around 1800RPM with my fans, it would make sense that the system would take so long to hit the fan thresholds. Since I had already altered the Critical fan speed thresholds down to 1050RPM, that's the only explanation that fits. I did try altering the Critical fans speed thresholds down to 0RPM for the PSU fans in the BMC but had the same result as before. I had to settle for using the original fans in the PSUs. This wasn't too much of an issue because they seem to be quieter than the other fans. Still, a lot of work to end up right where you started. A side note. I used the GEN II motherboard so your mileage may vary for GEN I and III models.

    ReplyDelete
  14. I won't be trying it for awhile but I just read about how you can alter the fan speed ranges with fan speed controller. This may be my only option to quiet down the PSU fans more.
    (http://www.silentpcreview.com/forums/viewtopic.php?t=37909&)

    ReplyDelete
  15. I downloaded a sound meter to my cell phone. I have no idea how accurate it is but I'll let you be the judge of that. I also picked up some original 2950 fans as spares so I've metered those too for comparison.

    Original Dell fans
    initial boot up jet engine test 1ft 76db
    idle from 1ft 57db. 2ft 52db. 3ft 49db.
    load from 1ft 55db. 2ft 50db. 3ft 48db.

    Low speed fans.
    Initial boot up jet engine test 1ft 73db.
    idle from 1ft 45db. 2ft 40db. 3ft 37db.
    load from 1ft 48db. 2ft 45db. 3ft 43db.

    I should note a few things. The Dell fans never leave there idle fan speed no matter the load they are put under. The low speed fans rev up to deal with the increased heat load but they do so very slowly. It is not a very noticeable process. Also, the low speed fans only stay in there elevated RPM state for about 90 to 120 seconds before they go back down to their idle states.

    ReplyDelete
  16. A 47 ohms resistor fixed the jet engine fan noise for me. I have to solder 5 each-47 ohms resistor inline to the red wire of each fan including the power supply. My Poweredge 2950 III is the SFF [small form factor} version with 8-2.5" drives. I put a 32GB RAM on it for my Hyper-V which also added more heat internally. Just be aware that the fan speed requirements for the 3.5" system might be different due to the amount of heat that the larger drives are generating. Please monitor the CPU temp if you have the 3.5" drives. Uptime now is 21 days with no issues. The server is in a 24u rack 6 ft behind me and I can work in my home office for as long as I want to without any noise complaint. There are no F1 or Fan Failure at POST, server booted to W2012 with no issues.

    Here's the link to my full report.
    http://blindcaveman.wordpress.com/2013/08/23/problem-dell-poweredge-2950-iii-jet-engine-fan-noise/

    ReplyDelete
  17. For those of you that are still looking for a solution:
    I have just completed a fair amount of bench testing of two different fans that are typically found in the PE2950 II and III series servers. The first being a Delta PFC0612DE and the other being a NMB-MAT 2415KL-04W-B86.
    During the bench test the fans would essentially stop and start continuously when placed in a circuit with greater than 33 ohms of resistance. The bench test was conducted with progressively more resistance (the exact values were 10,20,22,33,47,100 ohms). When placed over 33 ohms the starting and stopping behavior is hardly audible, however measuring current and voltage indicate that the fan does not operate well below 5.5V.
    Also, with a single 47ohm half watt resistor the temperature in two minutes was well over 280F with the fan dissipating heat from the resistor.
    My solution was to use a series of two 10 ohm, 10W resistors per fan (readily available at radio shack) which will allow more than adequate heat dissipation under a full load scenario without the risk of fire or damaging your equipment. I ran them for four hours on the bench and they were warm to the touch, and I ran them for four hours in the server and they were cool to the touch.
    Essentially this configuration will reduce the voltage to ~6V@~333mA.
    I'll have a sound level meter in the office, which we use to verify our equipment operates under OSHA guidelines, in a few weeks and will let you know how loud the server is with the original fans and how loud the server is with the fan modification at varying distances. The server is at tolerable level in my home lab as well with me just a few feet away, I can close the closet door if I need a bit of extra silence.

    ReplyDelete
    Replies
    1. Mate, were you able to measure the decibels on the unit after the resistor fix. I am planning for the same. Would really appreciate if you could share the noise level.

      Delete
  18. Being the proud owner of a 2950 III, I was thinking of doing a similar tweak.

    Instead of going for a static resistor. Could we consider using a potentiometer. this way we could freely adjust the resistance for the best airflow/noise ratio

    ReplyDelete
  19. I had initially considered a rheostat or potentiometer, but the price of a rheostat that would provide adequate heat dissipation under full load with <25ohm resistance was too high (~$25/rheostat).

    ReplyDelete
  20. Just completed this mod on a Dell 2950.
    I modified the BMC (v.2.50.00) firmware, and have a minimum RPM of 975, which you can see in Open Manage.
    I did not replace the fans - Instead, I used 68 ohm resistors in parallel (34 ohms resistance) - which also cuts the current through each in half (no more getting warm). You've got to stuff them in there, but they work just fine.

    I see some cycling of fan RPM, but it is so quiet that I don't notice it.
    Per OpenManage, they are spinning at 3100-3900 rpm range.. they seem to be cycling in that range, probably due to a big decrease in current.
    I have memory in "low power" mode.
    System board is at 31 C.
    The system pulls 227 watts, down from 230-235 before modifying the fans.

    This brings my system down to 53db outside the case. Before, I could hear it across the house.

    ReplyDelete
  21. Thanks to everyone for all this useful information - I'm having the same problem (trying to run a 2950 flat out, i.e. 8 cores at 100%, on several humanitarian distributed computing projects). My machine is in a normal domestic garage, well boxed in and with the doors shut, and I can hear it from inside my house.

    To clarify one point: If the supplied fans are completely unplugged, and I take care of the cooling requirements myself (i.e. with some other fan+heatsink arrangement), does the system still run? Or does the BIOS/firmware complain if the fans are determined to be 'missing/failed'?

    I think I'll try the 47-ohm resistor solution first.

    ReplyDelete