Updating The File Server to FreeNAS 11

Published: September 21, 2017 2017

Over the weekend I took the time to upgrade my file server from FreeNAS 9 to FreeNAS 11 (version 10 was an unreleased clusterf*ck). The big challenge wasn’t upgrading FreeNAS in place for the first time (I’ve done past upgrades by installing FreeNAS on new 16GB SanDisk Cruzer Fit flash drives). That part went quite smoothly. The big challenge was changing something that could jeopardize my data. By replacing the six-drive RAIDZ3 vdev with three two-drive stripped mirror vdevs, I had to destroy my data, restore my data from backup, and hope that my data wasn’t corrupted in the process.

The advantage of having a six-drive RAIDZ3 vdev is being able to lose three hard drives while still accessing the data (albeit with degraded performance until replacing the drives). While I never had multiple hard drives fail at the same time, I’ve had multiple hard drives fail one at a time from the oldest to the youngest. No sooner did the new drive finished rebuilding, the next drive starts signaling its imminent failure. The disadvantages of this configuration are not being able to recognize additional space until all six drives get replace with higher capacity drives, not being able to add another six-drive vdev, not getting the best read performance from a single vdev, and rebuilding a replacement drive takes forever.

The advantages of having three two-drive stripped mirror vdevs is being able to recognize additional space after upgrading two hard drives in a vdev, adding additional two-drive vdevs to increase overall storage, increasing read performance from multiple vdevs, and rebuilding a replacement drive at a faster rate. Since I’m using six out of eight hard drive slots inside my case, I need a SATA-III controller card and two 1TB Western Digital Red NAS 3.5″ drives to add another stripped mirror vdev. The disadvantage of this configuration is losing all data if two drives in a single vdev fail at the same time, which is more risky than losing more than three drives at the same time.

For my purposes, expansion and performance outweighs the risk of failure.

Before I could break down the old vdev to build the new vdevs, I had to ensure that my ~700GB of data got backed up to the Red Hat Linux box and the gaming rig. Backing up to the Red Hat Linux box took only a few minutes via rsync as this was the primary backup for the file server. Backing up to the gaming rig took longer as everything got copied over for the first time, taking over two hours at a 98MB per second transfer rate.

After the upgrade of FreeNAS got finished, I used the GUI to break down the existing configuration. Unfortunately, the GUI didn’t allow me to create stripped mirror vdevs. I had to do this from the command line: the create command created a two-drive stripped mirror vdev from hard drives /dev/ada0 and /dev/ada1, the two add commands created and added two more two-drive stripped mirror vdev from hard drives /dev/ada2 to /dev/ada5, and the status command displays the new configuration.

[creimer@oberon ~]$ zpool create storage mirror /dev/ada0 /dev/ada1
[creimer@oberon ~]$ zpool add storage mirror /dev/ada2 /dev/ada3
[creimer@oberon ~]$ zpool add storage mirror /dev/ada4 /dev/ada5
[creimer@oberon ~]$ zpool status storage
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada1    ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
          mirror-2  ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada5    ONLINE       0     0     0

Restoring the data from the Red Hat Linux box took over seven hours at a 35MB per second transfer rate. The slow transfer speed comes from rsync calculating the checksums on a $25 AMD AM1 quad-core processor that is less powerful than the $50 AMD AM3 dual-core processor on the file server or the $100 AMD AM3 eight-core processor on the gaming rig. While it would have been faster to restore the data from the gaming rig, the secondary backup without the checksums may have been less reliable than the primary backup.

What changed with the new configuration? Not much. I still have 6TB of raw space and 3TB of usable space as before. The only noticeable change is copying large video files between the file server and gaming rig at a 108MB per second transfer rate (a 10% increase). I’m sure FreeNAS 11 has some significant features over FreeNAS 9, but I haven’t noticed those yet.

New Mail Carrier Didn’t Get The Memo

January 26, 2015 2015

If you lived in an apartment complex for a long time, say, nearly ten years, you get used to how certain things get done. The mail carriers, for example, never deliver the packages to the door. Most people aren’t home during the day and packages left at the door tend to walk. Unlike the apartment complex I previously lived at for a brief period of time (six months), the leasing office doesn’t sign for packages. The mail carriers puts the package inside the lock box or return to the post office, leaving behind a lock box key or pick up slip in the mailbox. This month a new mail carrier who didn’t get the memo.

I’m in the process of rebuilding my FreeNAS file server by replacing parts round robin style. I started off with a NZXT Source 210 case that can hold eight hard drives, moving everything out of the old Dell Inspiron 570 that I inherited from my late father in 2012. This case had a single 120mm fan and mountings for six more 120mm fans. I subsequently ordered a pair of DeepCool 120mm fans for the front mountings to blow air over the existing four hard drives. Despite being in a roomier case with larger fans, the file server was unstable unless I laid the case on its side and took off the side panel for the heat to escape.

Earlier this month I ordered a CompTIA Security+ certification book and a bar of orange hand soap from Amazon. I got a text notification that the package got delivered but it didn’t say which carrier delivered it. After coming home from work that day, I found no lock box key or pickup notice in the mailbox. No package at my apartment door, which was more likely with Amazon and meant that the package had walked. I waited a few days to see if the tracking information was incorrect. Still no package. I notified Amazon early Saturday morning and the replacement package from Las Vegas arrived 12 hours later.

I ordered another pair of 120mm fans from NewEgg. Like the Amazon package the week before, I got a text notification that the package got delivered. Not exactly. The package got sent from the east coast via UPS and transferred to the post office for local delivery. (NewEgg normally ships my orders from their west coast location.) The website tracking showed no movement from the post office. For the next several days, I waited for a lock box key or a pickup notice inside my mailbox. Still no package.

I contacted UPS, who told me that the package was no longer their responsibility after handing off to the post office, and NewEgg, who told me to wait a few more days for the post office to get its act together. A week after UPS handed off the package to the post office, I filed a claimed with NewEgg and requested a refund for the “lost” package. On the day I got the refund from NewEgg, I got a text notification that the post office delivered my package.

Still no lock box key or pick up notice in my mailbox. No package at my door. I noticed a NewEgg box sitting at the door down the hallway. This door typically gets Amazon boxes. I checked the address label on the box—and it was my package. Correct apartment number on the package, but left at the wrong apartment door. Now I had to go through the trouble of returning the refund to NewEgg, as I was keeping the package now that I got it.

Stepping inside my apartment, I noticed a pickup notice stuck to the backside bottom of my door. When I first moved into my apartment, the door had a half-inch gap at the bottom for envelopes and insects to come through. The electric company had an energy efficiency campaign a year later that got the leasing office to install weather-stripping on all the apartment doors. Somehow the mail carrier shoved the pickup notice through the weather-stripping. I came to the conclusion that the mail carrier was new to the job, and filed a complaint with the postmaster general.

As for the file server, I put the new fans into the top mounts to blow hot air out. Five fans didn’t improve things by much. The hard drive at the very bottom of the case was consistently overheating to make the file server unresponsive. This hard drive was also seven-years-old and overdue for replacement, which was the last thing on my to-do list. I moved the fan from the second top mounting to the bottom mounting to blow in cooler air. The hard drive stopped overheating. The CPU temperature ticked up a few degrees from the new flow of hot air. Another fan from NewEgg for the second top mounting should do the trick.

Not sure if the new mail carrier got the memo on delivering packages.

The Black-and-White of The 2014 Mac Mini

Published: October 24, 2014 2014

2014 Mac Mini ExplodedApple’s announcement of the new Mac mini last week was an immediate disappointment for those of us waiting two years for updated hardware. With the level-entry Mac mini corresponding to the lower hardware specs of the level-entry MacBook Air, and, being like the MacBook Air, unable to upgrade the hard drive and memory after purchase, was a deal killer for most folks. Since the CPU fan for my vintage Black MacBook (2006) died this past summer, I need an affordable replacement system. Apple failed to deliver this year, but Other World Computing (OWC) has better options that fit the bill.

OWC sells refurbished Macs and various upgrades to extend the usefulness of each Mac. The cheapest option would be a 17″ iMac (2006) for $149 USD. This is the same generation as my Black MacBook, compatible with my existing hardware (i.e., memory modules, OWC 120GB SSD and external 250GB FireWire drive), and should boot up the SSD in an external closure as if it was yesterday. However, I’ll still be stuck with an unsupported 32-bit CPU that won’t run the newest Mac OS X, Yosemite, which was also announced last week.

The best option is a White MacBook (2010) for $449 USD. The unibody design has a back cover that comes off after removing ten screws, exposing the battery, DVD drive, hard drive, and memory for easy replacement. I’ll be able to pimp out this White MacBook more than I had with the Black MacBook.

  • Battery — The first battery for the Black MacBook lasted six years, and the second battery (manufactured in 2007) lasted two years. The battery for a 2010 laptop will need replacement sooner rather than later. Unfortunately, this item can only be obtained from the Apple Store for $129 USD.
  • DVD Player — OWC sells a data doubler adapter for $35 USD to replace the DVD player with a SATA hard drive or SSD. I’ll install the SSD from the Black MacBook to serve as the primary boot drive. OWC also sells an external optical drive enclosure to continue using the DVD player.
  • Hard Drive — My FireWire drive has two 120GB partitions for backing up the SSD with SuperDuper!, and a 10GB partition as a programming scratch pad. Since the White MacBook doesn’t have a FireWire connector, the internal 250GB hard drive will replace it. As prices for the larger capacity SSDs drop over time, I’ll replace the hard drive.
  • Memory — The White MacBook has 4GB RAM installed, which is fine for another year or two. The 2010 model maxes out at 16GB RAM, twice as much as the 2009 model. OWC sells a pair of 8GB memory modules for $200 USD.

All this expandability in a compact system for slightly more than the 2014 Mac mini with AppleCare and sale taxes included (~$650 USD). I got eight years of useful life out of the Black MacBook. I expect the White MacBook to provide at least four years, perhaps longer with a 64-bit CPU. More bang for the buck.

Reviving The Windows Gaming PC

Published: August 1, 2014 2014

The Apple Store revived my vintage Black MacBook (2006) several years ago after the CPU fan started screaming like a banshee, replacing the CPU fan, battery and keyboard. I hoped to get another six years of usage before getting a replacement system. Alas, the CPU fan started acting up several months ago. The system would shut down in 15 minutes after starting up. I could no longer use it to look for a new job, or, after being unemployed for nearly eight months, get it repaired or replaced. I had to switch over to my Windows gaming PC, which spontaneously reboots whenever I needed to do something.

Like most users who switched from Windows to Mac, I only turned on my PC to play video games. The last rebuild was in 2007 to upgrade the motherboard, CPU and memory for Windows Vista. That system was quite stable. A few years ago I replaced the dual-core processor with a quad-core processor and the video card from an ATI Radeon 3870 to an ATI Radeon 6790. That system wasn’t quite as stable. Upgrading to Windows 7 and Windows 8 over the years didn’t help much.

Was the quad-core CPU that came out years after the motherboard got manufactured and enabled with a BIOS update incompatible? Was the video card defective? Was the power supply failing in a mysterious way? Or was it all of the above?

Troubleshooting the PC was never urgent as I rarely played video games after getting serious about writing and suffering bouts of unemployment from my non-writing tech job. With the MacBook out of commission, I needed another computer system to continue my job search. The easiest solution was switching over to an old Dell system. However, I never take the easiest path if a harder—more educational—path is available.

After opening the PC and the Dell to lay side-by-side, I started switching out the video cards. With an old Nvidia Quadro video card in the PC and the 6970 video card in the Dell, both systems ran without problems. I then started checking the power requirements for the video cards and looked up the specs on the power supplies in each system.

The PC still had the power supply from 2007 with 20A on the 12V rail, but the Dell had a newer power supply with 40A on the 12V rail. Most new video cards required at least 25A on the 12V rail. The 6970 needed the extra juice for graphic-intensive applications. The solution became obvious. I switched the power supplies and put the 6970 back into the PC. (I didn’t bother putting the Quadro back into the Dell since the motherboard had a built-in AMD 4200 video chipset.) After wiping the hard drive and re-installing Windows 8.1, the PC was no longer spontaneously rebooting.

It didn’t take long to get the PC up and running with email to resume my job search. A few days later, I landed a new job. The only Mac-specific applications that I’m missing from the PC are Photoshop CS3 and Bento for ebook publishing. I can boot up the MacBook to complete any tasks within 15 minutes before it shuts down. Despite transferring operations over to my PC, I’m going to save up to get a replacement Mac later this year. Like most users who switched from Windows to Mac, the Mac is the better computing device.

MacWorld Expo 2014

Published: March 28, 2014 2014

MacWorld Expo 2014After scoring a FREE floor pass for MacWorld/iWorld Expo 2014 from MacKeeper (thanks!), I went up to San Francisco to check things out. I’ve been to the 2006 and 2007 expos when Apple was still the centerpiece exhibitor, and the 2009 expo when exhibitors shifted away from the Mac to the iPhone (the name “iWorld” got added in 2011). Crisscrossing the North Hall floor of the Moscone Center for two hours, my impression of the post-Apple expo haven’t changed at all since 2009: “Meh…”

When Apple was still at the expo, I dropped 50+ business cards into the fish bowls and brought home two plastic bags of promotional swag (most of which ended up in the trash). This time around I dropped two business cards into fish bowls, scanned my badge for two contests, and took home several promotional fliers. On the bright side, I’m not going to get 50+ phone calls from sales people when I interrupted their sales pitch to ask if I won anything, found out that I had no interest in buying their product, and then hang up.

The seventh anniversary of the iPhone introduction at the 2007 expo came earlier this year when I still haven’t gotten an iPhone. That changed several weeks ago when I picked up an iPhone 5c at the Valley Fair Apple Store. What finally pushed me over the edge to get an iPhone was finding a solution to my job search problem: I had phone numbers on my dumb cellphone and email addresses on my MacBook from recruiters, but no easy way to put the two together into a unified contact list. The iPhone brought the two together and LinkedIn provided all the recruiter-related info.

The only thing that caught my attention at the expo was a waterproof iPhone case by Optrix inside a five-gallon fish tank. The iPhone video outputted to a big screen monitor above the fish tank to demonstrate the underwater clarity. The person demoing the product conceded that the water was cloudy (looks like dust from unwashed gravel), showing me the crystal clear video taken from underwater in a swimming pool. I would love take videos of my tropical fishes from inside the tank as they’re very camera-shy if take pictures and videos from outside the tank.

The Metreon has undergone quite a few changes since my last visit when I came up to the City to see “The Dark Knight Rises” in July 2012. Target occupies the second floor, AMC Metreon 14 on the third floor, and the fourth floor, where I saw “Titanic—The Exhibition” in 2006, remain reserved for special events. The circular food court got cut in half to three restaurants, and all the other restaurants now had street entrances for people to get food without wandering through the building. The double cheese bacon hamburger at Super Duper Burgers was incredible for lunch, and the best part of going to San Francisco.

Sony Exits The eBook Market

Published: February 7, 2014 2014

Sony eBook ReaderI wasn’t surprised by the announcement that Sony is leaving the ebook market that it pioneered. Their main problem has always been the premium price that they demanded for their brand name. When I got hired for a six-week contract as a quality assurance test lead in 2005 to work on what would become the Sony Reader in the United States, my manager informed me that the initial price would be $600 USD. That was more than a Sony PlayStation 2. If I couldn’t afford a video game console, I wasn’t getting a dedicated ebook reader.

I was leading a group of ten testers in a conference room to review 300+ English ebooks on Japanese hardware with kanji characters. The engineers in India converted the PDF files from the publishers into HTML files. My group skimmed the ebooks and report any issues. (We weren’t supposed to read the ebooks, but I read every Star Trek novel that came my way and let the testers read if they met their quota for the day.) The engineers in Japan would refine the process. The cycle continued until we ran out of ebooks. I then summarized all our findings in a report on my last day.

I didn’t rush out to get a Sony Reader when it first came out the following year. For the price of a dedicated ebook reader, I got a Mac mini that was far more useful. Almost a decade later I still don’t own a dedicated ebook reader. I read Kindle ebooks on my iPad 2. My next ebook reader could be an iPhone 5C since my cellphone is due for an upgrade and I need a replacement for my first generation iPod Touch that has a battery will no longer hold a charge.

While Sony rested on their brand name, the rest of the industry innovated on technology and price. When I think of dedicated ebook readers, I think of Amazon and Barnes & Noble. Sony doesn’t even make it on my list as a distant third. Sony is the perfect example that being first to market isn’t an advantage, especially if the first mover isn’t motivated to maintain a leadership position.

As an ebook publisher, I’ll be removing the Sony links from my author website in the near future. As Smashword’s Mark Coker mentioned in his blog post, ebook sales to Sony was flat for the last four years while other ebook retailers had significant growth. I had a few ebook sales during the same time period. Since Sony is transitioning their readers to Kobo, my Smashwords ebooks will still be available. With Sony out of the way, the drumbeat for Barnes & Noble to get out of the business will get louder.

The Macintosh Came Out 30 Years Ago

Published: January 24, 2014 2014

Byte Magazine MacIntoshUnlike the first-generation iPhone in 2007, I wasn’t there for the introduction of the first-generation Macintosh in 1984. I was in the eighth grade at John Steinbeck Middle School in San Jose. According to the girls at school, I came from a “poor” family because my parents couldn’t afford cable TV to get MTV. We were too poor to own an Apple II. My parents gave me a Commodore VIC-20 for the Christmas the year before. When I informed my teacher that I got a computer, I got laughed out of the Apple II programming class in the seventh grade because he called the VIC-20 a toy (which it was).

A real computer, I learned, requires big bucks.

As my interests in computer programming and electronics developed in 1984, I read everything I could get my hands on. Byte Magazine was my primary source of information, where I first read about the Macintosh. The two most influential books I read that summer was a technical book on the Motorola 68000 processor that the Macintosh used, and “Hackers: Heroes of The Computer Revolution” by Steven Levy. I felt frustrated because I didn’t have a real computer to do anything with and the computer revolution was marching on without me. Never mind that I was only 15-years-old at the time.

I got a Commodore 64 for Christmas that year. Although a toy compared to the Apple II and Macintosh, this Commodore 64 was the first of three I would use for word processing, programming and video games over the next ten years. The Commodore 64 got me through the four bad years when I stayed home from high school and four good years at San Jose City College when I got my associate degree in general education.

The first Macintosh computer I used was a Macintosh Classic II at the SJCC library. English literature instructors demanded that all papers be turned in as either typewritten or laser-printed. The near letter quality (NLQ) setting on my dot matrix printer was barely tolerated. I would print out papers at home, re-type the papers into the Macintosh at the library, saved the file to a 3.5″ floppy, walked over to the checkout counter, insert the floppy into the Macintosh connected to the laser printer, and printed out the pages at ten cents a page.

As I worked in Silicon Valley, my experience with the Macintosh was touch-and-go in the Windows-centric corporate environment. Every time a co-worker taught me how to do something new on the Macintosh, I would get laid off from work two weeks later. Recruiters always teased me about Apple jobs but never submitted my resume because my work experience was—and still is—predominately Windows.

After I started earning the big bucks, I got a Mac mini in 2005 and a black MacBook in 2006. I later gave the mini to a friend who needed a Mac more than I needed an extra system. I’m still using the MacBook eight years later. Although suitable for word processing and web browsing, it’s no longer suitable for compiling programs in the background. I’ll be getting a replacement system later this year to commemorate the 30th anniversary of the Macintosh.

The iPhone Came Out Seven Years Ago

Published: January 10, 2014 2014
iPhone @ MacWorld 2007

Where were you seven years ago when Apple introduced the first-generation iPhone to the world?

My friend and I were attending MacWorld Expo 2007 in San Francisco. We did not, however, attend the keynote where Steve Jobs introduced the iPhone. We were poor Apple fanboys with FREE expo passes. We knew something big got announced. We heard plenty of speculation while standing in line to pick up our badges. Until the doors to the expo floor opened, we didn’t know how big the announcement was.

The centerpiece of the expo was an iPhone under a cylinder display case that rotated for everyone to see from all angles. As poor Apple fanboys, we had to wait. Professional photographers and video cameramen for the news media formed around the display case. The tech bloggers rushed in with their cameras. After various strata of Apple nobility took their pictures, we poor Apple fanboys got a chance to gawk at the iPhone.

We crunched the numbers and determined that we couldn’t afford an iPhone in the near future. Even if we could afford the iPhone, paying for the data plan on top of the regular cellphone service was a steep price to pay. Like hyenas watching the lions feast from a distance, we could only wait our turn to own an iPhone someday.

Seven years later, we still can’t afford to get an iPhone.

Not that we’re still poor Apple fanboys. I’m still using my first-generation black MacBook (2006), a first-generation iPod Touch from 2008, and an iPad 2 from 2012. My friend has an assortment of iPads. Although the entry-level iPhone 5C is affordable at $99 USD, the monthly service charges for a two-year contract are still sky-high. That’s hard to justify in an uncertain economy that never seems to end.

Blowing Up A Sex Doll In Space

January 3, 2014 2014

[youtube url=http://www.youtube.com/watch?v=G17RdYm8x2Q]

During the eighth grade in 1984, I had a junior engineering class to draw blueprints and build models. The big projects we did were a cut-away section of a house, a racing car propelled by a carbon-monoxide cylinder, and a tissue paper hot air balloon.

My favorite project was the hot air balloon. I glued alternating sheets of green and white tissue paper into eight panels, cut each panel into a balloon-shape pattern from a template based on a mathematical formula, and glued the panels together to form a 48″ balloon with a narrow neck.

Because I assembled my balloon at home, the cats got to it with their sharp claws. I had to glue on 234 patches inside the classroom while everyone else flew their balloons. The best-case scenario for my balloon was for it to fill up and fall over due to the extra weight. The worst-case scenario was for it to catch fire from an ember (which did happen to some balloons, including one in flight).

The teacher held the flue over the wood-burning trashcan to direct the hot air. I held the balloon neck over the flue opening to fill it up. After I tied off the neck with a rubber band and let it go, the balloon floated 20 feet into the air and out of the courtyard to everyone’s astonishment. A group of us chased after the balloon two blocks down the street from the school. That’s the furthest any balloon went that day.

The teacher announced at the end of class that my balloon was a kludge—something that worked when it wasn’t expected to work. That was the happiest day for me in the eighth grade.

Kids today don’t know how good they have it when doing engineering projects. A weather balloon filled with hydrogen can carry a mounted camera to transmit video of a test object going into space. Like the video of this inflatable sex doll that went up 102,000 feet, exposed to extreme gamma radiation and the low surface pressure of Mars, and crashed somewhere in the Nevada desert.

Gives new meaning to the old Star Trek engineering motto: “She’s gonna blow, captain!”

Getting Slimed At Work

July 31, 2013 2013

After coming back from lunch at my non-writing tech job, I noticed a pair of heavy-duty hoses running from the men restroom, up a ladder in the hallway and into the plenum space above the dropped ceiling. I carefully stepped past the hoses and warily watched the ceiling tiles as I made my way back to my desk. I remembered the last time the AC units got worked on and rubber hoses ran through the plenum space at a different company, where someone almost got slimed at work.

While working as a video game tester at Accolade/Infogrames/Atari (same company, different owners, multiple identity crises) in the late 1990’s, I became the intermediary between the Q.A. and I.T. departments when the company was still called Accolade at the Stevens Creek Boulevard office in San Jose. A long-running love/hate relationship had developed between the testers and the techs over the years. The biggest complaint then was that the techs ran a Diablo game server that slowed down the entire network during the lunch hour. The testers can’t read Blues News if the network crawled like molasses.

Management decided to upgrade the network infrastructure by replacing the network operating system (NOS), Netware (which didn’t allow more than 254 computers) with Windows NT Server (which was surprisingly stable for a Microsoft operating system), and changing the physical network from 10Base100 (10Mb over thin coaxial cable) to 100BaseT (100Mb over twisted pair cable) to increase the available bandwidth.

Since the techs weren’t welcome in the Q.A. department, my job was to upgrade all the computers with the new network cards and plug them into the wall. If a network issue went beyond the wall, I had to walk over to the I.T. department and try to convince them that the problem was on their end.

One day I was talking to the I.T. manager about something. His work area in the corner of a narrow room had a Viewsonic 21″ monitor. A very big, very nice and very expensive monitor. CRT monitors like that cost about $2,000 USD in 1998. A few years later, I would get a Viewsonic 19″ monitor for $400 USD. Since I wasn’t welcome to stand inside the tech room, I stood outside the doorway to talk him.

We were, in fact, talking about his new monitor when something started banging around in the plenum space to violently shake the ceiling tile. He pushed himself away from his desk to roll back in his chair. The ceiling tile broke into big chunks that gave away as green goo slimed his new monitor. We both got out of there. I left him alone as he had a complete meltdown over his ruined monitor.

The maintenance crew transferred coolant fluid between AC units on the roof by using an ordinary garden hose that had a weak spot that ballooned with green goo before exploding. Not sure why the hose through the plenum space and not on the rooftop. As for the I.T. manager, he went back to his old 17″ monitor.