Breaking files by hand
The workshop makes its glitches with code, but it all started the other way round: open a file with the wrong program and see what happens. This is what I learnt from stAllio!'s primers, Phillip Stearns and James Connolly's talk.
What a glitch is
A glitch is a fault you can see or hear: the image that only half downloads, the tape that crinkles, the game drawing garbage when the cartridge isn't seated properly. Glitch art causes them on purpose and keeps what comes out.
It isn't new. Nam June Paik was putting magnets on television sets in the sixties to bend the picture. What computers changed is that the error stopped being an accident of the machine and moved inside the file: you can save it, repeat it and share it.
The fun is that you're never fully in control. You change four bytes and the result depends on how the format is built inside. That's why half of this craft is learning what files are like.
Before glitch: net.art
In the mid-nineties a handful of artists started using the web as material rather than as a shop window: the browser, HTML, frames, broken links and errors were the work. It was called net.art, with a dot, like a file name.
Legend has it the name came from an anonymous email Vuk Ćosić got in 1995, so mangled by incompatible software that the only readable bit was "net.art". True or not, it's already a glitch story.
Many of these works are still online and can be visited as they are; others have been restored by Rhizome so they run in today's browsers.
- jodi
- wwwwwwwww.jodi.org (1995): a page that looks like spewed-out code and, if you view the source, hides drawings. Breaking the web on purpose, right from the start.
- Olia Lialina
- My Boyfriend Came Back from the War (1996): a love story told with HTML frames that keep splitting every time you click.
- Vuk Ćosić
- Besides naming it, he turned classic films into characters in ASCII History of Moving Images (1998): the grandfather of the ASCII workshop.
- Alexei Shulgin
- Form Art (1997): works made only of buttons, checkboxes and form fields, the greyest bits of the web.
- Net Art Anthology
- Between 2016 and 2019 Rhizome restored and annotated a hundred net art works, from the nineties to now. The best place to start.
Who's who
Not a complete list, just the people who keep turning up in what I've been reading, roughly in order of appearance.
- Nam June Paik
- In 1965 he put a magnet on a television and the picture twisted (Magnet TV). He's the grandfather of all this: video as a material you can bend.
- Steve Rutt and Bill Etra
- In the early seventies they built the Rutt/Etra video synthesizer, which turns the picture into lines that rise with the light.
- jodi
- Joan Heemskerk and Dirk Paesmans. Since 1995 they've made websites and games that look broken on purpose: code, errors and the screen as the work.
- Cory Arcangel
- He opened a Super Mario Bros. cartridge and removed everything but the clouds (Super Mario Clouds, 2002). Hacking the machine is glitch too.
- Takeshi Murata
- Monster Movie (2005) is one of the works that made datamoshing known: a video that melts once its keyframes are gone.
- Rosa Menkman
- Dutch artist and theorist. She wrote the Glitch Studies Manifesto (2010) and the book The Glitch Moment(um) (2011): why the error says something about the machine and about whoever is looking.
- Kim Asendorf
- In 2010 he published ASDFPixelSort, the Processing sketch that made pixel sorting fashionable.
- stAllio!
- His databending primers (from 2008) taught many people to break files by hand, starting with the WordPad effect.
- Phillip Stearns
- Breaks digital cameras and image files, and weaves the results into blankets and tapestries (Glitch Textiles). Keeps a huge list of resources.
- Don Miller (NO CARRIER)
- Makes NES cartridges designed to cause glitches live, like glitchNES, and plays them at gigs.
- GLI.TC/H
- The Chicago glitch festival (2010–2012), run by Nick Briz, Jon Satrom, Evan Meaney and Rosa Menkman among others. It brought half the glitch world together.
Why some files break better than others
Almost every image file has two parts: a header at the start that says what it is (size, colours, tables), and then the image data. Break the header and the program won't open it any more. Break the data and it opens and draws it wrong, which is what we're after.
In an uncompressed BMP each byte is directly part of a pixel's colour, so a change stays where you made it: noise, bands, colours bleeding. In a JPG the data is compressed and each chunk depends on the one before: a single changed byte drags the error through the rest of the picture, with the colour shifts and blocks you see in the workshop.
In a JPG, the real data starts after the FF DA marker (the 'Start of Scan'). Better leave everything before it alone.
With a hex editor
The most direct way: open the file and see its bytes, one by one, in hexadecimal. Any editor will do; in the browser there's hexed.it, and on Windows the classic is HxD.
- Always work on a copy: you're going to break it.
- Open it in the editor and find where the header ends. In a JPG, look for FF DA and skip a few more bytes.
- Change some bytes in the data part, or copy a chunk and paste it somewhere else. Start small: ten changes already show.
- Don't write FF in a JPG: it opens a new marker and the image will usually stop opening.
- Save and open it. If it won't open, go back to the copy and change less, or further towards the end.
With an audio editor (sonification)
A file is just bytes and an audio program doesn't care where they came from: tell it they're sound and it treats them as sound. That way you can put an echo, a reverse or a phaser on a photo, and the effect shows up as bands and repeats.
It works best with uncompressed images: BMP or TIFF.
- In Audacity: File → Import → Raw Data. Choose 'U-Law' or '8-bit unsigned', one channel.
- Select everything but the start (that's where the header is; a small chunk is enough) and apply an effect: echo, reverb, reverse, pitch shift…
- Export as 'Other uncompressed files', header 'RAW (header-less)', with the same encoding you imported with.
- Change the extension back (.bmp) and open it. If it won't open, try again leaving more of the start untouched.
Try it in the workshop →Glitch Art Resources (Phillip Stearns) ↗
The WordPad effect
stAllio!'s first primer describes a trick that became famous: open a BMP image in WordPad, the Windows word processor, and save it. WordPad thinks it's text and 'fixes' some bytes along the way (line breaks, mostly), so everything after shifts a little and the image comes out crossed by diagonal stripes.
Windows 11 no longer ships WordPad, but you can imitate the effect: any program that converts line breaks (0A into 0D 0A) in a file that isn't text does the same.
GIF
A GIF carries its palette, 256 colours at most, near the start. Change palette bytes and the image doesn't break: it changes colour all over, as if developed with different chemicals.
The data is LZW-compressed, and there one change turns everything after it into streaks and garbage. With an animated GIF you can break just some frames. Part three of stAllio!'s primer goes through it byte by byte.
Try it in the workshop →Databending primer, part 3: the GIF ↗
Video: datamoshing
A compressed video doesn't store every frame whole. It stores a complete one every so often (the keyframe, or I-frame) and, in between, only what moves (the P-frames). Delete the keyframes and the motion of the new shot is applied on top of the old shot's picture, and the two melt together: that's datamoshing.
Takeshi Murata took it into museums in the mid-2000s and then it reached music videos. It used to be done with old versions of Avidemux and Xvid videos; today there are tools like FFglitch, built on FFmpeg.
With the machine: consoles, cables and circuits
You can also break things from the outside. Circuit bending opens up cheap toys and gadgets and joins points on the circuit that were never meant to touch. A half-inserted cartridge in an old console draws impossible graphics.
Don Miller (NO CARRIER) made glitchNES, an NES cartridge designed to cause glitches live with the controller, and plays it at gigs.
Real glitch or fake glitch?
Iman Moradi, in his thesis Glitch Aesthetics (2004), set 'pure' glitches, the ones that just happen because something fails, apart from glitch-alikes: the ones sought out or imitated on purpose. Almost all glitch art is the second kind, and that's fine.
Even among the deliberate ones there are differences. Really breaking the file (changing bytes, deleting frames) gives results you can't predict. A filter that imitates the error (VHS lines, shifted colours) always gives the same thing, and has become just another look, for ads and music videos.
The workshop has both and doesn't hide it: the broken JPG really breaks the file; VHS and RGB channels are imitations. Both are useful; it's just good to know which is which.
Glossary
The words that come up all the time, one sentence each and, if the workshop does it, a link to try it.
- Glitch
- A brief fault you can see or hear. It comes from the slang of radio engineers and 1960s NASA; it existed before computers.
- Databending
- Breaking a file by opening or editing it with a program that isn't meant for it: a photo in an audio editor, a BMP in WordPad.
- Compression artefact
- What a compressed format leaves in plain sight: JPG blocks, bleeding colour, video turning into squares when the connection drops.
- Macroblock
- The 16×16-pixel squares compressed video splits each frame into. In a datamosh they're what moves and drags the old picture along.
- Keyframe (I-frame) and P-frame
- An I-frame is a whole frame; P-frames only store what changed since the last one. Remove the I-frames and you get a datamosh.
- Datamosh
- Removing a video's keyframes so the motion of one shot paints itself over another.
- Pixel sorting
- Sorting a row or column's pixels by brightness or hue, usually only along a stretch. The picture looks like it's dripping.
- Chromatic aberration
- Colours that don't line up: red, green and blue shifted, as in a cheap lens or a misaligned projector.
- Dithering
- Faking more colours than there are by mixing dots of the few left. It's what 3- and 8-bit computers did.
- Slit-scan
- Building a picture from slices taken at different moments: whatever moves stretches. It's the trick behind the stargate corridor in 2001: A Space Odyssey.
- Scan lines and tracking
- The lines of a CRT television and the band crawling up the screen when the VCR can't read the tape properly.
- Sonification
- Treating data that isn't sound as sound, to apply audio effects to it.
- Circuit bending
- Opening an electronic gadget and joining points on its circuit so it does what it shouldn't.
- Zalgo
- Text covered in Unicode combining marks stacked above and below the letters.
- Mojibake
- Text that comes out garbled because it was read with the wrong encoding: 'ñ' instead of 'ñ'. The word is Japanese.