Husband | Father | Retro Gamer | Remote Cloud Engineer @demandbase | PGP 5F70 F2AC BD58 F580 DF15 3D1F 4FA2 70F5 CD36 71F | (views are my own)
43 stories
1 follower

Writing and Running a BBS on a Macintosh Plus

1 Share

In 2015, I wrote a custom BBS server in Ruby and had been using it to run the Kludge BBS on a small OpenBSD server in my home office since then.

Last year after writing a lot of C on my Macintosh Plus, I had the itch to write a new BBS server so I could move the BBS to run on another Mac Plus. As with all software development projects, it took quite a bit longer than expected, but last month I finally got far enough with the development to deploy the new BBS on a Mac Plus.

Table of Contents

  1. Hardware
  2. Revision Control
  3. Threading
  4. MacTCP
  5. Console
  6. Design Detours
  7. Screen Saver
  8. Web Login over WebSockets
  9. File Transfers
  10. Telnet Bots
  11. Future Work
  12. Summary


My deployment target is a platinum Macintosh Plus with an 8 Mhz Motorola 68000 processor, 4 Mb of RAM (the maximum supported), and a SCSI2SD v5.5 external SCSI hard drive with a 2 GB HFS “disk” on a Samsung PRO Endurance microSD card. The Mac is running System 6.0.8.

Noctua fan

While the Mac Plus is normally fanless, I was concerned about heat buildup running it 24/7, so I installed a small Noctua fan inside the case that still operates at near silence. With the internal CRT also powered on 24/7, I thought it would be useful to add a toggle switch to reduce power usage and heat, and avoid screen burn-in, but this does not seem very straightforward. I did recap the analog board before setting everything up and it’s been running for a couple months now without issue, but time will tell how long everything lasts.

For network connectivity, the Mac Plus has a DaynaPORT SCSI/Link-3 SCSI Ethernet adapter which connects to a managed switch on its own VLAN, connected to my OpenBSD firewall which routes the Mac Plus’s static IP over my internet connection. TCP/IP is provided by MacTCP 2.1.

Side note: I implemented a virtual DaynaPORT SCSI/Link-3 Ethernet adapter in the PCE MacPlus emulator, allowing it to do real network connections through a tap device on the host.

For modem connectivity, I am using a US Robotics 5686 modem connected to a Grandstream HT814 VoIP ATA, which routes SIP traffic through the same OpenBSD firewall running Asterisk, which then routes that traffic through Twilio for PSTN service. I took the US Robotics modem board out of its normal case and rigged it up inside the case of an Apple Modem 300/1200, and soldered a green power LED to the front.

For backups and for transferring files between my Mac Plus workstation, I am using an Iomega Zip 100 SCSI drive. For those keeping track, that’s a SCSI ethernet device, SCSI hard drive, and SCSI Zip drive on an 8 Mhz computer from 1986. I don’t know why the non-server PC market never caught on to using SCSI for hard drives and peripherals.

DaynaPORT SCSI/Link-3 Ethernet, Iomega Zip 100 Drive, and Apple 300/1200 Modem

Revision Control

As with all of my classic Mac OS projects, I would be writing the code for the BBS on a Mac Plus in the THINK C 5 IDE. Since THINK C’s editor only has one level of undo, once a file is saved, I can’t go back without restoring a file from a backup, and my previous method of making a date-stamped copy of my code directory was not scaling well.

Before I started writing the BBS, which I am calling Subtext, I wanted to have a proper revision control system so I could commit things in small chunks, view diffs of what’s in my work directory, and if needed, revert changes to a file.

There are some proprietary development tools that work on newer machines running System 7, but since I am doing everything on System 6 and am limited to 4 MB of RAM, I opted to create my own lightweight tool that I call Amend.

I also created a lighthearted GitHub-like web interface to Amend repositories, which I call AmendHub. I periodically upload the Amend repo from my Mac Plus to my OpenBSD laptop, and then upload it to AmendHub where new amendments are imported.

Continued development of Amend along the way also took quite a bit of time, like having to change its on-disk database format, but at this point I consider it to be quite stable.


The next hurdle was needing a cooperative threading mechanism. In the same way that my Ruby BBS used Fibers, this would allow me to write code in a very top-down, procedural format without having to use annoying callbacks, while still allowing concurrent interactive user sessions. This took a bit of research and development on classic Mac OS since I had never built such a thing before, but ultimately resulted in a small uthread library (code) that uses setjmp and longjmp to give each thread its own stack and then switch between them once a thread calls uthread_yield.

In Subtext, this is done whenever the user’s session has to wait for input or wait for its per-node module to flush its output.


Some other BBS packages that I’ve come across for System 6 and 7 rely on the Communications ToolBox for doing TCP/IP communication and modem access, but I decided not to go this route.

Subtext is written to use the ubiquitous MacTCP system extension for TCP/IP access which is a bit quirky compared to Unix sockets, especially when implementing servers. A TCP stream is created to passively listen on a TCP port, but once a connection is established on it, a second stream must be immediately setup to listen for a second connection or that request will be lost.

There are also some quirks related to finding the state of a listening stream that were not documented and took a lot of real-world debugging to figure out why the server would just stop accepting new connections after a day or so. Luckily the amount of bot-scanning traffic that the BBS receives just being connected to the internet provides a lot of garbage traffic to experiment with.


Since Subtext runs on a Mac with a screen, it seemed appropriate to allow the sysop to log in locally on a console. This required writing a VT100-capable terminal client which could now be easily extracted out into a dedicated telnet client. I’m currently using ZTerm on my Mac Plus workstation to connect to other BBSes via modem, but it doesn’t support telnet without a Serial->WiFi adapter.

Design Detours

Winding along its 8 month development, I had a few detours and over-engineering of obscure components. While I’m most likely the only one that will ever use Subtext, I avoided hard-coding anything specific to my BBS in it. I made a templating mechanism to dynamically expand variables in views (like {{ username }}, and a GUI-based view editor that can be used for creating things like the menu:

While I initially intended to add GUI-based editors for things like the user database and message boards, once I had a working sysop console and could access all of the BBS functionality locally, I ended up writing this into Subtext as interactive menus. This has the benefit that I can manage them remotely through telnet rather than having to be at the console.

Screen Saver

Since the Mac Plus will always have its screen on, I wanted to avoid CRT burn-in while it’s just showing the log window waiting for a call. I can’t rely on another screen saver like After Dark because it doesn’t know anything about the BBS activity and can’t automatically deactivate when there is activity that I want to see. It’s also very CPU-intensive to make toasters fly, so I want it stopped as soon as a new connection comes in.

I wrote a simple opportunistic screen blanker that kicks in at a configured timeout of inactivity when there are no active sessions, and just blacks out the entire screen for a small amount of configured time or until something is logged.

Web Login over WebSockets

Once telnet and modem connectivity were complete, I wanted to bring back web access. My Ruby BBS had an integrated web server which served some custom JavaScript that talked back to the BBS over a WebSockets connection. My JavaScript implemented a VT100-capable terminal so the BBS could send out ANSI escape sequences just as if it were talking to a telnet client.

While the Mac Plus is capable of serving telnet clients on its own, I was not going to implement the whole WebSockets spec in addition to a custom HTTP server. Additionally, due to MacTCP’s single-connection-on-a-socket limitation, it would be difficult to make the HTTP server not drop HTTP requests during the listen->open->listen state change.

Since my OpenBSD firewall is already in between, I setup nginx with websocketd to execute a stripped-down telnet binary on the OpenBSD firewall.

When a new connection comes in, nginx serves the front-end HTML and JavaScript. The JavaScript makes a WebSockets connection back to nginx, which then passes it to websocketsd, which launches telnet. The telnet binary connects to the Mac Plus through its normal telnet server, but passes the web client’s IP address as a telnet environment variable, which Subtext will honor and display it as the remote IP it’s talking to, even though it’s actually communicating with the firewall’s IP. websocketsd automatically handles the shuffling of input and output data between the telnet client and the WebSockets connection.

Since the previous HTML displayed a fake Windows 3.1-era telnet client, I changed it to resemble a Mac running ZTerm. The old interface used my DOS 437-codepage font to display high-ascii (“ANSI”) box characters properly in the browser, but since the new BBS is only using plain ASCII and is being displayed on a fake Mac screen, I needed a TrueType version of the old Monaco 9 font from System 6. I couldn’t find a proper one, so I made one by creating a BDF-format font with all of the characters from a screenshot, then converted it to TrueType with Bits’N’Picas.

Fake Telix web interface to old BBS, fake ZTerm web interface of new BBS

File Transfers

The last major part of the BBS was handling file transfers with ZMODEM. The ZMODEM protocol is rather large and has a complicated state machine, so I wasn’t looking forward to writing my own implementation, especially for something that has to be compatible with other terminal software. The few open-source C implementations available have questionable licensing or are GPL’d, but I eventually found a BSD-licensed version from Tera Term that was easy to separate.

Integrating it with Subtext was not terribly difficult but it did require a week of debugging with other terminal software to fix bugs and set timeout limits appropriately. There is still no ZMODEM support in the web front-end, so file transfers don’t yet work there, and I’m sure there will be other minor compatibility issues that need work, but to be honest, the file areas were barely used at all on the old BBS.

Telnet Bots

Once the new BBS was deployed last month, I started to get annoying bot traffic. Thousands of bots and infected machines constantly scanning the internet are a minor inconvenience to a modern server, but are troublesome to a very slow server. Every few minutes of every day, a random IP tries to login to my BBS with default usernames and passwords like “root”, “sysadmin”, “enable”, etc. If the connection is closed, they will try again, over and over as it runs through its dictionary. Since the Mac is very limited in resources, these connections can prevent legitimate traffic from connecting or make their sessions very slow since the BBS is using cooperative threading.

Subtext has a list of banned usernames and when a login is attempted with one of these, the connection is dropped and the IP is banned. Future connections from that IP will immediately close, but unfortunately there is no way to refuse the connection before it opens so the BBS still has to do its dance of accepting the new connection, then starting a new listening socket, then closing the first connection.

A recent change goes a step further and sends a UDP packet to my OpenBSD firewall containing the IP to be banned, and a small Ruby server running there adds the IP to a pf table, immediately blocking all further IP access from the bot.

Future Work

I haven’t implemented SSH logins yet, which would also forward from the OpenBSD firewall. For my old Ruby BBS, it ran a small, custom Go-based SSH server. When a user connected and provided a username and password over SSH, the Go SSH server would connect to a local socket that the BBS was listening on and send the username and password, the BBS server would authenticate it and respond with an acknowledgment or failure, and if it succeeded, the Go SSH server would open a new TCP connection to the BBS, proxying data between the BBS and the client.

I still need to implement this, but using the telnet client in-between. The telnet protocol makes it easy to send data out-of-band, such as the already-authenticated username. However, since SSH-scanning bots are more prevalent and aggressive than the occasional telnet-scanning bot, I’m not terribly motivated to write this new SSH server.


If you read this far and want to try out the BBS, you can telnet to, call +1 312-654-0090 with a modem (8N1), or login with a fancy web browser at If the BBS doesn’t answer, please try again later as it might be seeing a large spike in traffic (or has crashed and I’m scrambling to find the bug).

The freely-licensed C source code for the Subtext BBS server is available on AmendHub.

If you’re interested in classic Mac development, check out my C Programming on System 6 videos or chat with us in the #cyberpals IRC channel on Libera Chat.

Read the whole story
640 days ago
San Diego
Share this story

Old and Small Technology

1 Share

Old technology is any tech that’s, well… old. Small technology is any tech that has a small footprint: doesn’t require a powerful machine to run, doesn’t have a lot of bloat, doesn’t have anti-features like spyware and tracking. Technology that is old enough is almost always small because, by modern standards, that’s all that was possible back then. Some small tech is old, some is modern. Embedded systems are an example of modern small tech development in many cases.

Read the whole story
662 days ago
San Diego
Share this story

Malls Arent Actually Dying

1 Share
Read the whole story
664 days ago
San Diego
Share this story

The Shape of Rome – Ex Urbe

1 Share

via-dei-fori-imperiali-3The new Mayor of the city of Rome, Ignazio Marino, just announced his intention to destroy one of the city’s central roads, the Via dei Fori Imperiali, and turn the area around the old Roman Forum into the world’s largest archaeological park.  Reactions have ranged from commuters’ groans to declarations from classicists that this single act proves the nobility of the human species.

This curious range of reactions seems the perfect moment for me to discuss something I have intended to talk about for some time: the shape of the City of Rome itself.  We all know the long, rich history of the Roman people, and the city’s importance as the center of an empire, and thereafter as the center of the memory of that empire, whose echo, long after its end, still so defines Western concepts of power, authority and peace.  What I intend to discuss instead is the geographic city, and how its shape and layers grew gradually and constantly, shaped by famous events, but also by the centuries you won’t hear much about in a traditional history of the city.  The different parts of Rome’s past left their fingerprints on the city’s shape in far more direct ways than one tends to realize, even from visiting and walking through the city.  Rome’s past shows not only in her monuments and ruins, but in the very layout of the streets themselves.  Going age by age, I will attempt to show how the city’s history and structure are one and the same, and how this real ancient city shows her past in a far more organic and structural way than what we tend invent when we concoct fictitious ancient capitals to populate fantasy worlds or imagined futures.  (As a bonus to anyone who’s been to Rome, this will also tell you why it’s a particularly physically grueling city to visit, compared to, say, Florence or Paris.)

9780521609104cvr_red.qxdSigmund Freud had a phobia of Rome.  You can see it in his letters, and the many times he uses Rome as a simile or metaphor for psychological issues, both broadly and his own.  He fretted for decades before finally making the visit.  Part of it was a cultural inferiority complex.  Europe’s never-fading memory of the greatness of the Roman empire was intentionally magnified in the Renaissance by Italian humanists who set out to convince the world that Roman culture was the best culture, and that the only way to achieve true greatness was to slavishly imitate the noble Romans.  Italians did this as a power play to try to overcome the political weakness of Italy, but as a result, in the 19th and 18th centuries, many intellectuals in many nations were brought up in a mindset of constantly measuring their own nations only by how far they fell short of the imagined perfection of Rome.  Freud was one of many young intellectuals in Germany, Poland, and other parts of Europe who were terribly intimidated by the Idea of Rome, and the sense that their own nations could never approach its greatness.

But Freud had a second fear: a fear of Rome’s layers.  In formal treatises, he compared the psyche to an ancient city, with many layers of architecture built one on top of another, each replacing the last, but with the old structures still present underneath.  In private writings he phrased this more personally, that he was terrified of ever visiting Rome because he was terrified of the idea of all the layers and layers and layers of destroyed structures hidden under the surface, at the same time present and absent, visible and invisible.  He was, in a very deep way, absolutely right.  Rome is a mass of layers, the physical form of different time periods still present in the walls and streets, and when you study them enough to know what you are really looking at, they reach back so staggeringly far, through so many lifetimes, that if you let yourself think seriously about them it is easy to be overwhelmed by the enormity of it all.

I will begin by discussing a single building as an example, and then the broader structure of the city.

The Basilica of San Clemente:

San Clemente is a modestly-sized church a couple blocks East of the Colosseum, one of many hundreds of churches in Rome, and, in my mind, the most Roman.  It was built in honor of Pope Clement I (d. 99 AD), an important early cleric who traveled East and returned, making him one of the most important linking figures between the Catholic and Eastern Orthodox worlds.  One enters the church from a plain, hot street populated by closed doors plus an antique shop and a mediocre pizzeria.  Outside the door is a beggar disguised as someone who works for the Church trying to extort money from tourists by convincing them that they have to pay him to enter.  Within, a lovely, lofty church with marble columns, frescoed chapels, a beautiful stone floor, stunning gold mosaics in the nave, and a gilded wood ceiling.  It is populated by milling tourists, and perhaps a couple of the Irish Dominicans who are now its custodians.   It is reasonably impressive, but when we pause and look more closely, we realize the decoration is not as simple as it seems.  Nothing matches, for a simple reason: No two pieces of this church are from the same time.


The basic structure of the church, the actual edifice, is from the twelfth century.  But nothing else.

Look at the columns first: beautiful colored marble columns with delightful translucent swirls of stone.  But they don’t match: they’re different colors, even different heights, and have non-matching capitals and different size bases to try to make them fit.  These columns weren’t made for this building, they are looted columns, carried off from Roman buildings all around the city and repurposed for this Church.  These columns, therefore, were cut about 1,000 years before the construction of this church.

San Clemente Detail (2)

The floor too is Roman mosaic tile, inlaid with pieces of porphyry and serpentine, materials unachievable after the empire’s fall.  If they are here, they were carried here after the 12th-century Church was built and re-used.


What else?  There is the stunning mosaic.  It looks like nothing else we’ve seen in Rome, and with good reason.  It looks Russian or byzantine, a totally different style.  Foreign artists must have come in to create this, not in a Roman style of decoration at all but one more Eastern.  Our Eastern Church devotees of Saint Clement have been here.



We turn around next, and spot a lovely side chapel with frescoes of a saint’s life, in a familiar Renaissance style.  We might have seen this on the walls of Florence, produced in the late 1400s or earlier 1500s, and can immediately start playing Spot the Saint.

Roma, Basilica San Clemente in Laterano

But next we make the mistake of looking up, and realize that this massive hanging gilded wood ceiling is entirely wrong, with overflowing ribbons and a dominant central painting of a much more flowy, ornamented, emotional, voluptuous Baroque style than everything else.  The artist who painted those modest Spot the Saint frescoes would never drown a scene in little cherubs and clouds like this, nor would that ceiling ever have been near these Roman columns.

basilica san clemente ceiling

The upper walls too have Baroque decoration. Even an untrained eye is aware something is wrong.  The practiced eye can tell instantly that the ceiling must be late sixteenth century at the very earliest and is more likely seventeenth or eighteenth, three hundred years newer than the Spot the Saint frescoes, which were two hundred years after the mosaics, which are two hundred years after the church was built using stolen Roman materials that were already 1,000 years old.  Freud, exploring the church with us, has vertigo.

Next we look down.

San Clemente Detail

What’s this?  What are these arches in the wall next to the floor?  Why would there be arches there?  It makes no sense.  Even in a building that used secondary supporting arches in the brickwork there would be a reason for it, a window above, a junction, and they would end at floor level.  Our architecture-sense is tingling.

So we go down stairs…


Welcome to the 4th century Roman basilica which the 12th century upper church was built on top of.  Here we see characteristic dense, flat Roman bricks, and late classical curved-corner ceiling structures laying out what used to be an early Christian church.  This church was 800 years old when it was buried to build the larger one above it.  The walls are studded with shards of Roman sculpture, uncovered during the excavations, bits of broken tombs, halves of portrait faces and the middle of an Apollo, and a slab with a Roman pagan funerary inscription on one side which was re-used and has an early Christian inscription on the other side, in much cruder lettering.

San Clemente

And here too there are frescoes.  Legend has that Saint Clement’s remains were carried from the East back to Rome in 869 AD, and this lower church is the place they would have been carried to, as we see now in a fresco depicting the scene, painted  probably shortly thereafter.


Other 9th century frescoes (300 years older than the church above) show the lives of other now-obscure figures who were important in the 800s.  One features a portrait of an early pope (Leo IV), the only known image of this largely-forgotten figure.  Another features Christ freeing Adam from Limbo, and to their left a man in a very Eastern-looking hat, another relic of the importance of this church as a center for Rome’s contact with the east.


Another wonderful fresco, of the life of a popular hermit, features a story in which a pagan demands that his servants carry the saint out of his house, but he goes mad and believes a column is the saint, and flogs and curses his slaves as he forces them to carry the column.  In this fresco we find inscriptions in Latin, but also a phrase coming out of the man’s mouth (a very crude one cursing his slaves as bastards and sons of prostitutes) which is the oldest known inscription in a language identifiable as, not Latin, but Italian.  The Italian language has come to exist between the construction of this church and the construction of the one above.  (The inscription is at the bottom in the white area above the column, hard to make out.)


You can see it better in this reconstruction:


One more fresco is worth visiting: the Madonna of the funny-looking hat.

Madonna and Child in 4th-cent Basilica San Clemente

When archaeologists opened up the under layer, they found a Madonna, probably 8th century, which then decayed before their eyes (horror!) due to exposure to the air.  Underneath they found another Madonna (delight!) wearing this extremely strange hat.  They looked more closely: the Christ child in her lap is not original, but was painted on after the Madonna.  This is not a Madonna at all, it is a portrait, and that hat belongs to none other than the Byzantine Empress Theodora.  Someone painted a portrait of the empress here (who used to be a prostitute, I might add), then someone else redid her as a Madonna, then, a century or two later, someone else painted over that Madonna with another Madonna, now lost, who presumably had a more reasonable hat.

Wandering a bit we find more modern additions, post-excavation.  One of the most beloved 20th century heads of the Vatican Library has been buried here, just below the now-restored old altar of the lower church.  And the tomb of St. Cyril [or possiby it contains Cyril and his brother Methodius – there is debate] is here.  They are the creators of the Glagolitic alphabet (ancestor of the Cyrillic), surrounded by plaques and donations and tokens of thanksgiving from many Slavic countries who use that alphabet.  Below is a modern mosaic, thanking them for their work:


And nearby there are stairs down…   Freud needs to stop and breathe into a paper bag.

There are stairs down because this is not the bottom layer, not yet.  The 4th century church was built on top of something else.  We descend another floor and find ourselves in older, pre-Christian Roman brickwork.  We find high vaults, frescoed with simple colorful decoration, as was popular in villas and public buildings.  Hallways and rooms extend off, a large, complex building.  Very complex.  Experts on Roman building layout can tell us this was once a fine Roman villa of the first century AD.  In that period it had sprawling rooms, a courtyard, storerooms… but its foundations aren’t quite the right shape.  If we look at the walls, the layout, it seems that before the villa there was an industrial building, the Mint of the Roman Republic (you heard me, Republic!  Before the Empire!), but it was destroyed by a fire (the Great Fire of 64 AD) and then rebuilt as a Roman villa.  Before it was a church… before it was another church.


Except… there are tunnels.  There are narrow, meandering tunnels twining out from the walls of this villa, leading in strange, unpredictable directions, and far too tight to be proper Roman architecture.  This villa was on a slope, and some of these rooms are dug into the rocky slope so they would have been underground even when it was a residence.  Romans didn’t do that.



Houston, we have a labyrinth, a genuine, intentional underground labyrinth, and with a bit more digging we find out why.  This was a Mithraeum, a secret cult site of the Mithraic mystery cult, which worshipped the resurrection god Mithras.  Here initiates dwelled in dormitories for their years of apprenticeship, waiting their turn to enter the clandestine curved vault, sprawl on its stone couches, and participate in the cult orgy in which they take hallucinogens, play mind-bending music, and ritually sacrifice a bull and drink its blood in order to achieve resurrection.

s clemente1

We wander still farther, daring the labyrinth, much of which has not yet been excavated, and come upon another room in which we hear the bubbling of a spring.  A natural spring, miraculously bubbling up from nowhere in the depths of Rome.  Very probably a sacred spring.


While Freud sits down to put his head between his legs for a while (on a 1st century AD built-in bench, I should add) we can finally piece this muddle of contradictory and mismatched objects together into a probable chronology:

SanClemOnce upon a time there was a natural spring bubbling up at this spot in what was then the grassy outskirts of early Rome.  It is reasonable to guess that a modest cult site might have sprung up around this spring, honoring its nymph or some such, as was quite common.  In time, the city expanded and this once-abandoned area became desirable for industrial use as the Republic gained an empire.  The Republic’s Mint was built here, making use of the convenient ice cold water, and likely continuing to honor its associated spirit.  Decades pass, a century, two, Rome expands still further, and chaos raises an Emperor.  After the Great Fire of 64 AD, it becomes convenient to move the Mint out of what is now a desirable central district of the expanding city, so the site is purchased by a wealthy Roman who builds his house here.  Decades pass and the builder, or his son, is converted to the exciting cult of this new god Mithras who promises his followers, not the gray mists of Hades, but resurrection and eternity. Since he is wealthy, he converts his home to the use of the cult, and digs tunnels and creates the underground Mithraeum.  For a generation or two this villa hosts the cult, but then Constantine comes to power and a new cult promising an even more inclusive form of salvation comes into vogue.  The villa, which is now three hundred years old, is buried, a convenient architectural choice since the ground level of the city has risen several times due to regular Tiber floods, so the old house was in a low spot.  A new church is built on top, and serves the Roman Christians of the local community for a few generations.  The fall of Rome is usually marked at the first sack by they Visigoths in 410 or the sack by the Vandals in 455, but the conquerors are also Christian so the church stands and still serves the neighborhood, though its population is much smaller.  Now the main Emperor moves to the East, and in the 500s, when the church is about 200 years old, someone paints a portrait of the empress on the wall, then a generation later someone else decides a Madonna is more appropriate, and puts a baby in her lap.  Two or three more generations go by and Cyril and Methodius bring the bones of Clement from the East, and they are buried here, a great day for the neighborhood!  Commemorated with more frescoes.

basilica-di-san-clemente-servus-servorum-dei-basilica-di-san-clemente-37443Another century, two, we are well into the Middle Ages, and this old Roman building is old-fashioned and very low since the ground level has risen further.  The local community, and devotees of St. Clement, decide to build a new church.  They loot columns and flooring from other Roman sites, and bury the old church, producing the 12th century structure above, but using the walls of the older one as the foundation, so the arches still show in the walls.  The new church is very plain, but is soon decorated using mosaics provided by Eastern artists who come to visit Clement and Cyril.  After a few generations the Renaissance begins, and we call in a fashionable Florentine-style artist to fresco one chapel.  A few centuries later Pope Clement VIII comes to power and decides to spiff up San Clemente, initiating the internal redecoration which will end with the ornate baroque ceiling.

DSCN9436Oh, and somewhere in there someone slapped on a courtyard on the outside in a Neoclassical style, because it became vogue for buildings to look classical, so we may as well add a faux-classical facade onto this medieval building which we no longer remember has a real classical building hidden underneath.  Not long after the Baroque redecoration is begun, the nineteenth-century interest in archaeology notices those arches in the walls, and starts digging, re-exposing the lower layers.  Devotees of St. Cyril and lovers of history, like the head of the Vatican Library, begin to flock to San Clemente as an example of Rome’s long and layered history, and so it gains more layers in the 20th century as donations and burials are added to it.  Every century from the Republican Roman construction of the Mint to the 20th century tombs is physically present, actually physically represented by an artifact which is still part of this building which has been being built and rebuilt for over 2,000 years.  Not a single century passed in which this spot was not being used and transformed, and every transformation is still here.  And all that time, from the first sacred spring, to the Mithraism, to today’s Irish Dominicans, this spot has been sacred.

This is Freud’s metaphor for the psyche: structure after structure built in the same space, superimposing new functions over the old ones, never really losing anything.

This is Rome.

San Clemente is exceptional in that it has been largely excavated and is accessible, but every single building in Rome is like this, built on medieval foundations which are built on classical ones.  I can’t tell you how many times I’ve gone into a random pizzeria and found a Renaissance fresco, or a medieval beam, or Roman marble.  I’ve gone into a cafe restroom and discovered the back wall was curved because this was built on the foundations of Pompey’s theater (where Caesar was assassinated).  I’ve gone into churches to discover their restrooms used to be part of different churches.  Friends have this experience too.  During my Fulbright year in Italy I had a colleague who was studying Roman altars, half of which you could only get at by ringing the bell of strangers’ apartments and saying: “Hello!  I’m an archaeologist, and according to this list there’s a Roman sacrificial altar here?” to which the standard response is, “Oh, yes, come on in, it’s in the basement next to the washing machine.”  I have another friend who thinks he’s found a lost chapel frescoed by a major Renaissance artist hidden in an elevator shaft.  Another friend once told me of a pizza place with a trap door down to not-yet-tallied catacombs.  I believe it.

As with San Clemente, so for Rome: layers on layers on layers:

If San Clemente’s narrative starts with a sacred spring and the Roman Mint, Rome’s narrative starts with scared people on a hill.


Welcome to the archaic period.  You are a settler.  Your goals are securing enough food to stay alive, and avoiding deadly threats.  The major threats are (A) lions, (B) wolves, (C) wild boar, (D) other humans, who travel in raiding parties, killing and taking.  You are looking for a safe, defensible spot to settle down.  You find one.  The Tiber river, which floods regularly producing a fertile tidal basin rich with crops and game, takes a bend and has a small island in it.  At that same spot there are several extremely steep, rocky hills, almost like mesas, with practically cliff-like faces.  In such a place you can live on top of the hill but hunt, farm, and gather on the fertile stretch below.  And you can even sail up and down the river, making trade and travel easy.  Perfect.


The very first settlement at Rome, in the archaic period, was a small settlement on the Capitoline hill, one of the smallest hills but closest to the river.  (Are you, perchance, from a country?  With a government that meets in a “capitol” building?  If so, your “capitol” is named after the Capitoline hill, because that’s how frikkin’ important this hill is!)  The valleys around are used mainly for farming, but also for burials, and the first tombs are very simple ones, just a hole with dirt, or sometimes a ceramic tile lid.  The buildings in this era are brick decorated with terra cotta.  Eventually the first major temple is built on the Capitoline hill, with a stone foundation but still terra cotta decoration, and is dedicated to Jupiter. Its foundations remain, and you can see them, in situ, in the Capitoline museum which will be built on the same spot a few millenia later.

This hill turns out to be a great place to live, and the population thrives.  In time the hill is too crowded.  People spread to the neighboring hills, and start building in the little valley in between.  As the population booms and spreads to cover all seven hills, the space between the first few becomes the desirable downtown, the most important commercial center, where the best shops and markets are.  This is the Forum, and here more temples and law courts and the Senate House are built.

In time, defensive walls go up around the area around the hills, to make a greater chunk of land defensible.  In time, the walls are too constrained, so another set goes up around them.


As the population booms and Rome becomes a serious city, serious enough to start thinking about conquering her neighbors and maybe having a war with someone (Carthage anyone?), this area is now the super desirable downtown.  The commercial centers migrate outward to give way to monuments and temples, the Mint is built out on a grassy spot past where there is not yet a Colosseum, and the hills near the Forum become reserved for sacred spaces, state buildings, and the houses of the super rich.  On one, the Palatine hill, a certain Octavian of the Julii builds his house, and when Caesar is assassinated and the first and second triumvirates result in an Emperor, it becomes the imperial palace. (Does your capital contain a palace?  If so it’s named after the Palatine hill, because Augustus was so powerful that all rulers’ grand houses are forever named after his house).

Rome again spills over her walls and builds even farther out.   The great fire of 64 AD destroys many districts, but she rebuilds quickly, and what was the Mint is replaced by a villa which soon becomes a Mithraeum.  Rome reaches its imperial heights, a sprawling city of a million souls, and the seven hills that were once defensive are now sparkling pillars of all-marble high-class real estate, and also very tiring to climb.

With Constantine, Christianity now becomes a centerpiece of Roman life, and of the city’s architecture.  Major Christian sites are built: St. Peter’s, St. John Lateran, St. Paul’s Outside the Walls, etc.  These sites become pilgrimage centers, and economic centers.  They are scattered in far corners all around Rome, but all the sites have something in common: they are in corners.  The major Christian centers of Rome are all on its periphery, not in the center.  There are two reasons for this.

First, and simplest, the center of Rome was, by this time, already full.  Sometimes you could find an old villa that used to be a mint to build a small church on, but the center was full of mid-sized temples, which could be rededicated but not replaced, and huge imperial function spaces and government buildings, plus valuable real estate.  If you want to build a big new temple to a big new God, you need to do it in the not-yet-developed areas around the city’s edge.

Second, many of these sites were built on tombs, like St. Peter’s, built across the river in the cheap land no one wanted. Roman law banned burying the dead within the city limits, because disturbing a tomb could bring the wrath of the dead upon the city, but if you build immovable tombs in the middle of your city it makes city redevelopment impossible, so they have to be outside.  This is the origin of the necropolis or “city of the dead”, the cluster of tombs right outside the gates of a Roman city, where the residents bury their dead.  Some major Roman Roads, like the Via Appia, are still lined with rows of tombs stretching along the street for miles out from where the city limits used to be defined.  Thus early Christian martyrs were buried outside the city, and their cult sites developed at the edges of the city.  The land which became the Vatican, for example, was across the river, full of wild beasts and scary Etruscan tribesmen in archaic Rome, then was used for a necropolis in Imperial Rome, had enough empty cheap land to build a big circus (where much of the throwing of Christians to the lions happened, since only in such cheap real estate could you build a stadium big enough to hold the huge audiences who wanted to come see lions eat Christians), and finally Constantine demolished the circus and necropolis to build St. Peter’s to honor St. Peter who had been martyred in that circus and buried in the necropolis in secret 300 years before (when San Clemente was still a Mint).  St. Peter’s, and the other Christian sites, bring new importance to Rome’s outskirts.  We now have a bull’s-eye-shaped city, in which imperial government Rome is the center, and Christian Rome is a ring around the outside, with rings of thriving, happy commercial and residential districts in between.


410 and 455 AD: outsiders arrive and plunder the city.  Many thousands are killed, and the beautiful center of Rome is ransacked, temples toppled, looted, burned.  In the Forum, the raiders throw chains around the columns of one of my favorite layered Roman buildings, the temple of Antoninus and Faustina.  The Visigoths try to pull the columns down with their chains, and fail, but slice gouges deep into the stone which you can still see today.  To re-check time, the Temple of Antoninus and Faustina was built in 141 AD, when San Clemente was a villa with an active Mithraeum in it.  When it received these scars in the Visigothic raid, the Mithraeum had been buried, and the church built on top was just starting to be decorated.  And underneath the Temple of Antoninus and Faustina we have found archaic grave sites which were 1,000 years old when the temple was built 2,000 years ago–the people buried in those graves very likely drank water from the spring that still burbles up under San Clemente.  As for the Temple of Antoninus and Faustina, a few centuries after its near-miss, the temple will be rededicated as the major Roman church of San Lorenzo, due to a legend that it was on these temple steps that Saint Lawrence was sentenced to be grilled alive.  And not far from it, the Lapis Niger was excavated which contains a language which has not yet become Latin, much as San Clemente’s frescoes preserve one which is becoming Italian. One language evolved into another, then into a third, but this spot was still being used, just like today.

Rome was sacked, but afterwards Rome was still there.  The Goths didn’t just take everything and leave – the Ostragoths who followed the Visigoths decided to become the new Roman Emperors and rule Italy.  The surviving Roman patrician families started working for the new Gothic king, but still had a Senate, taxes, processions, traffic cops, and did all the early Medieval equivalents of keeping the trains running on time.  A century later, in the 540s, the Plague of Justinian hits and Rome loses another huge hunk of its population. But it still ticks on, and there is still a Senate, and a people of Rome.

So what was different?  From a city-planning sense, the key is that the population was much smaller.  In a sprawling metropolis designed to hold a million people, we now had maybe twenty thousand.   Thus, as always happens when a city’s population shrinks, real estate was abandoned.  But instead of abandoning the outskirts, people abandoned the middle.  Rome was important mostly as a Christian center now, with the pope, and pilgrims coming to major temples, so they occupied the edges, and that’s where the money was.  Rome becomes a hollow city, a doughnut, with an abandoned center surrounded by a populated ring.  We have reached Medieval Rome.  The city population lives mainly over by the Vatican, in the once empty district across the river, and a few other Christian sites around the edge.  The middle of the city has been abandoned so long that the Tiber has buried the ruins, and people graze sheep in what used to be the Forum.  The old buildings are now little more than quarries, big piles of stone and brick which we can steal from if, for example, we happen to need some nice columns to build a new church on top of this old church of San Clemente.

DSCN9360Enter the Renaissance, Petrarch, and humanism.  Petrarch writes of the glory that was Rome, and convinces Italy that, if they can reconstruct that, they can be great again, just as when they conquered the Goths and Germans.  Popes and lords become hungry for the symbols of power which Rome once was.  Petrarch reads his Cicero and his Sallust, and visits the empty center of the city.  This is the Capitoline Hill, he says, where once stood the Temple of Jupiter, and where the Romans crowned their poets and triumphant generals.  Wanting to be great again, the popes volunteer to rebuild the Capitoline, as do the wealthy Roman families, who sincerely believe they are descended from the same Roman Senators who kept the bread and circuses running on time through Visigoths and more.  Michelangelo and Raphael crack their knuckles.  New palaces are built on the Capitoline Hill, neoclassical inventions based on what artists thought ancient authors like Vitruvius were talking about.  In time the population grows, and Rome’s wealth increases thanks to the Church and to the PR campaign of Petrarch and his followers. The empty parts of the inner city are re-colonized, by Cardinals building grand palaces, and poorer people building what they can to live near the Cardinals who give them employment.  But it is all built out of the convenient stone that’s lying around, and on top of convenient foundations that used to be the buildings of Constantinian Rome when she boasted 1,000,000 souls.

Rome grows and refills and grows and refills from the outside in, with the Capitoline as a new center artificially reconstructed by Renaissance ambition.  As the 18th and 19th centuries arrive, the city is full again, but the middle ring, between outside and center, is all the newest stuff, to the historian and tourist the least interesting.  This is why everything that tourists come to see in Rome is a long bus ride from everything else, and why you have to go up and down a million exhausting hills to get anywhere.  Rome has a belt of cultural no-man’s-land in and around it, separating the center from the Christian outskirts, and making it forever inconvenient.

In the 18th and 19th centuries we also start to have archaeology, and dig up the Forum, and begin to protect and reconstruct the ancient monuments, and recognize that this largely abandoned patch of valley behind the Capitoline Hill is, arguably, the most important couple blocks of real estate that has ever existed in the history of the world.  We paint Romantic paintings of it, and sketch what it must have looked like once, and it becomes part of the coming-of-age of every elite young European to make the pilgrimage to it (that Freud so fears!) and see the relics of what once was Rome.  Everywhere else the classical layer is under a pile of palaces and churches and pizzerias, but here in the precious Forum valley, between those hills that sheltered the first Romans, we have lifted the upper layers and exposed Rome’s ancient heart.


A00175567Specifically Mussolini built a road straight through the middle of the Forum.  Fascism was a strange moment in human history, and Rome’s, and left a lot of scars.  One of them is the Via dei Fori Imperiali, a grand boulevard running along the Forum and around the Capitoline, which Mussolini built so he could have processions, and to declare to the world how sure he was that no one would care about the Roman relics he was paving over.  They would not care about the Temple of Jupiter, or the Renaissance palace on top of it, but about the new monuments he carved into the city’s heart. Those, and he, would be remembered, Caesar and Augustus forgotten.

To quote my favorite column by the old Anime Answerman: “Dear kid, please tell your friend that no one has ever been more wrong in the entire history of time.”

Mussolini, like the Visigoths, came but did not entirely go.  One of his remnants is a system of large boulevards scarred into the face of the city, intended for his grand Fascist processions.  Many of these are now difficult to eliminate, since car traffic in Rome is already a special kind of hell (fitting as a subsection of Circle 7 Part 2, I’d say, violence against ourselves and our creations, though it could be 4, hoarding/wasting, or yet another pouch of 8). The worst offender, though, is this road which is currently still covering up about a quarter of the ancient Forum, and also separates a quarter of the remaining Forum from the other half.  It is this road that the new Mayor proposes to eliminate.  The extra Fascist decoration which Mussolini added to the “wedding cake” will stay, the right call in my opinion, since Fascism is now one of Rome’s layers, just as much as the Visigothic scars on the Temple of Antoninus and Faustina.  But lifting the road away will give us the true breadth of the Forum back in a way no pocket diagram can replicate.  The transition will be painful for the FIATs and Vespas that now swarm where long ago the early Romans fought Etruscans and wild boar, but it is also an important validation of the Forum’s status as Rome’s most special spot. Everywhere else is layers.  Everywhere else, when there’s Baroque on top of Renaissance on top of medieval, we leave it there.  The altar stays behind the washing machine, and the need to open yet another catacomb is smaller than the need to have a working pizzeria.  But in the Forum the layers have been lifted away.  This one heart of one moment in Rome’s history, or at least one patch of about seven active centuries, we expose and preserve in honor of the importance that little spot has had as the definition of power, empire, war, and peace for Europe for 2,000 years.  Thus, I hope you will all join me saying thank you to Mayor Marino.

The Forum is our relic of Rome’s antiquity, but it is not, for one who knows the city, the true proof that this is a great ancient capital.  That would be clear even if not an inch of Roman marble remained in situ.  The proof of Rome’s antiquity is its layout, the organic development of a wildly inconvenient but rich city plan, with those impassable hills at the center, the Tiber dividing the main city from the across-the-river part which is still the “new” part and still politically distinct, with its own soccer team, even after thousands of years.  Antiquity is the nonsensical distribution of city mini-centers, the secondary hubs around the Vatican and St. John Lateran, the crowded shops clinging to the cliff-like faces of the hills, the Spanish Steps which are there because you have to go up that ridiculous hill and it’s really tall.  Antiquity is not the Colosseum, it’s the fact that the Colosseum is smack inconveniently in the middle of a terrible traffic circle, definitely not where anyone would put a Colosseum on purpose if the modern city planners had a choice.  Antiquity is structure, the presence of layers, unlike young, planned cities where everything is still in a place that makes sense because that city has only had one or two purposes throughout its history.  Rome has had many purposes: shelter, commerce, conquest, post-conquest/plague refugee camp, religious capital, center of cultural rebirth, new capital, finally tourist pilgrimage site.  All those Romes are in a pile, and the chaos that pile creates is the authentic ancient city.  Rome is that cafe bathroom with a curved wall that proves it is where Caesar was assassinated.  In another thousand years I don’t know what will be there, a space-ship docking station or a food cube kiosk, but whatever it is I know it will still have that curved back wall.

If you enjoyed this, see also my historical introduction to Florence.

FOOTNOTE:  For those who care, the context of that Anime Answerman quotation:

Kid writing in: “Dear Anime Answerman, my friend tells me that Inuyasha is a more violent show than Elfen Leid, and I don’t believe them, but I can’t tell them they’re wrong because my Mom won’t let me watch Elfen Leid.”

Answerman: “Dear kid, please tell your friend that no one has ever been more wrong in the entire history of time.”

Print Friendly, PDF & Email
Read the whole story
1100 days ago
San Diego
Share this story

Bare Metal K8s Clustering at Chick-fil-A Scale

1 Comment

by Brian Chambers and Caleb Hurd

At full scale Chick-fil-A will be running Kubernetes at the Edge in each of our 2000 restaurants. That means roughly 6000 devices at the Edge running Kubernetes.

One of the biggest challenges associated with this is bare metal clustering on-the-fly, in-restaurant.

While most Kubernetes deployments are in the cloud or benefit from skilled technicians that are physically located near their deployments (or at least equipped with remote access), our deployments are completed by installers who focus only on initial hardware installations. They never connect to the compute devices directly — rather they connect ethernet and power cords, and then look at an app to check the status of the cluster as it self-bootstraps. Replacements are completed by restaurant Owner/Operators or their teams, which are sometimes less technical.

On top of that, our Edge deployments are not exactly in a “datacenter environment”.

Edge Compute hardware and a typical installation

Clustering: Options we considered

To solve this clustering challenge, we surveyed the landscape and considered several options:

Kubespray — we started with the Ansible based Kubespray but found it to be fairly brittle. When things went well we got a cluster. When they didn’t, we created a brick that was hard to transform back into a computer. We also found that the process to initiate clusters was very slow, often taking as much as 30 minutes on our hardware stack. While we are loathe to discredit the project long-term as a result, this was enough for us to move in a different direction.

Openshift — it can create K8s clusters, but we didn’t like the idea of being closely tied to a vendor’s solution for a critical part of our infrastructure.

Kops — we are big fans of kops, and we use it to deploy our cloud “control plane” Kubernetes clusters. Unfortunately, when we began our Edge Compute journey, kops was not a viable bare metal solution. We look forward to seeing how it develops in the future.

Kubeadm — another Kubernetes proper clustering utility. It looks promising, but is definitely much more complex (perhaps due to its flexibility) than some of the alternatives, including…


In our first iteration, RKE was our winner. RKE is a Kubernetes clustering engine provided by Rancher Labs. While we decided NOT to use Rancher 2.0 to manage our clusters, we do like the simplicity of using RKE to initialize and maintain a cluster.

To use RKE, you need to determine a leader node and provide it with a configuration YAML file that includes data about the cluster, including the host names for the nodes that will participate in the clustering activity.

If nodes in the cluster are added or removed (or die), the configuration file needs to remain an accurate representation of the current and to-be member nodes. Failure to keep this configuration up-to-date will result in failed clustering attempts. While we believe the absence of a node should not fail the clustering initialization/update, that’s the way it works today.

Installation Process

Our installation process in restaurants is very simple to perform — unbox the devices, plug them into power and the labeled switch ports, and that’s it. They automatically boot on power, and they self-bootstrap and cluster. While this is awesome since it enables a non-technical user to execute installations and replacements without any knowledge of Kubernetes or even the overall architecture, it does create a need for a much more sophisticated bootstrapping process.

The to-be cluster nodes need to coordinate with each other to determine who is going to participate in clustering. They also need to elect a leader to execute the cluster creation via RKE.

Introducing Highlander

To solve this problem, we developed Highlander… because there can be only one. Cluster initiator, that is. One cluster initiator.

Highlander is part of our base Edge image. When each node boots, it UDP broadcasts its presence and asks if there is an established leader. It also begins to listen itself. After a few seconds with no replies, it will send another broadcast announcing that it has declared itself the leader. Are there any objections? Should nobody object, the node will soon establish itself as the cluster leader and respond to any future requests asking about an available leader.

If another node has already claimed the role of leader, the new node will acknowledge the declaration. The existing leader will “RKE up” to include the new node in the cluster.

The nodes gossip periodically to ensure the leader is still there. If the leader ever dies, a new leader will be elected through a simple protocol that uses random sleeps and leader declarations. While this is simple and unsophisticated, it is easy to reason about and understand, and it works effectively at scale.

Once leader election is complete, highlander also ensures the cluster is properly configured. In our environment, this includes:

  • Switching out KubeDNS for CoreDNS
  • Setting up Istio / other core control plane pods
  • OAuth identity negotiation **

**each of our nodes gets its own identity and a short-lived JWT for accessing authenticated resources. Highlander provisions this identity and makes the token available as a Kubernetes secret.

Process in aggregate

While we mostly focused on cluster initialization during this post, here is a look at our entire process for node initialization on-the-fly, in-restaurant.

(Inevitable) Failures

Infrastructure breaks and we want to be resilient to its failure. Node failures can occur for many reasons: device failure, network switch failure, power cords accidentally unplugged. In all of these cases, we have to quickly diagnose what is a true failure and what is an unrelated anomaly. That process is complex and the subject for another post in the future. That said, when we diagnose failures, our process is to drop ship a base image replacement to the restaurant (complete with visual installation instructions) and have the Owner/Operator or their team execute the placement.

Meanwhile, our Kubernetes cluster will continue to operate on a reduced number of nodes, and will be ready to welcome the replacement node when it arrives.

What’s next?

We like RKE, but there is still a chance that we will make a shift to a different clustering engine in the future. We would consider revisiting Kubeadm as it matures, or possibly consider rolling our own cluster manager to give us more control over the process. Having developed Highlander, it would not be a stretch to complete the rest of the clustering process. We will keep a close eye on the ecosystem and see what develops.

Like what we’re doing? Chick-fil-A is always looking for talented engineers to join our team. We would love to hear from you, so contact Brian Chambers on LinkedIn if you would like to hear about opportunities to join the team.

Read the whole story
2114 days ago
Next time I'm getting a gallon of sweet too I'll need to check out their #kubernetes edge nodes
San Diego
Share this story

The State of Building Images on Kubernetes

1 Comment

At KubeCon+CloudNativeCon in Copenhagen in May, many talks focused on the work required to build continuous integration and continuous deployment (CI/CD) pipelines using containers. One of the major issues still remaining in the container world is specifically that last bit of the CI/CD pipeline: building, storing, and securing containers built for internal software projects.

Steve Speicher, principal product manager on the Red Hat OpenShift Team, spent a good deal of time at KubeCon looking into the solutions and remaining pain points that exist around dynamically building and managing containers within a more traditional agile development environment. “A lot of people want to leverage Kubernetes for the build in the pipeline itself, and so that’s one of the things we’re talking about here and learning more about what people are interested into leverage the platform to do more CI/CD,” said Speicher.

“You have people who have build CI/CD farms and infrastructure and then their deployment platform,” added Ben Parees, principal engineer at Red Hat. “With Kubernetes and OpenShift you have the opportunity to put that all in one, so your cluster is both your build platform, your test platform, and your deployment platform. It’s easy to scale up multiple instances, to test them, run your builds there,” said Parees.

“Kubernetes is a platform to run images, but you are on your own to produce those images,” said Parees, highlighting a remaining gap in the workflow for many enterprises looking to adopt Kubernetes. “Now we’re starting to see people getting interested in the problem of ‘How do I build those images securely, reproducibly, and again, can I use the platform? Docker Build requires a lot of permissions that you may not want to give everyone to produce your images. It’s fine to do it on your workstation, maybe not OK on a shared cluster. So now we’re starting to see some additional tools coming out, things like the S2I tool, the Kaniko tool, and a few other tools around building images that give more flexibility and do it in a secure way on platform,” said Parees.

Reproducibility is a big deal for building container images, said Speicher. He said one of the values for a system like those developed by Red Hat is around patching routines. “You need to roll that out in a secure way,” said Speicher. “And you need to make sure when you do that, it’s a repeatable process. Through that, we have a centralized registry where you can have the content, and then you can have an image change trigger so it can notify the build whenever these images change. So when that happens, it’s going to happen at a pretty large scale, and it’s going to happen to a large number of applications, you want to make sure that system can automatically build those things and then reproduce the final deployable image in a way that’s consistent.”

In this Edition:

2:01: What are the frictions in CI/CD that are still hampering developers?
3:37: How are container images now being built for Kubernetes?
7:16: Where is the developer experience now when developers are starting to go out and build containers themselves, and what is the guidance as it progresses into Kubernetes distribution?
10:10: What does S2I produce?
15:14: What is the infrastructure Red Hat has built to allow for these more declarative environments?
19:20: Addressing the speed aspect in building single-image-layer applications with S2I.

Red Hat sponsored this post.

Feature image via Pixabay.

The post The State of Building Images on Kubernetes appeared first on The New Stack.

Read the whole story
2117 days ago
This is probably my number top request for #kubernetes right now.
San Diego
Share this story
Next Page of Stories