Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 03 20:26

    brunchboy on master

    Turn off broken "Edit this Page… (compare)

  • Oct 01 22:12

    brunchboy on master

    Add antora and netlify credits … (compare)

  • Sep 08 16:14

    brunchboy on master

    Update other dependencies. (compare)

  • Sep 08 16:04

    brunchboy on master

    Update CoreMidi4J. (compare)

  • Jun 11 03:19

    brunchboy on master

    Update to link to netlify hoste… (compare)

  • Jun 11 03:01

    brunchboy on master

    Update doc readme to reflect us… (compare)

  • Jun 10 21:40

    brunchboy on master

    Fix typo in function name. (compare)

  • Jun 10 21:18

    brunchboy on master

    Remove insecure HTTP link, than… Try simplifying API doc path, u… (compare)

  • Jun 10 20:48

    brunchboy on master

    Working on netlify doc build. (compare)

  • Jun 10 20:27

    brunchboy on master

    Update Clojure version. Update Codox version. Try setting up Netlify build fo… (compare)

  • May 27 18:14
    brunchboy edited #70
  • May 27 18:12
    brunchboy edited #70
  • May 27 18:12
    brunchboy labeled #70
  • May 27 18:12
    brunchboy opened #70
  • May 22 05:21
    brunchboy closed #69
  • May 22 05:21

    brunchboy on master

    Update jquery.mincolors color p… Start wedding reception show. (compare)

  • May 22 01:44
    brunchboy opened #69
  • May 22 01:42
    brunchboy closed #23
  • May 22 01:41
    brunchboy closed #65
  • Apr 29 21:08

    brunchboy on master

    Fix more broken images due to w… (compare)

James Elliott
@brunchboy
The place where I have gone furthest in working with CDJs is in beat-link-trigger which some producers and artists have been using to sync video walls with CDJ tracks in Resolume, and also to sync Ableton Live improv performances with CDJs using the Ableton Link protocol at music festivals.
Austin Wright
@awwright
Also I suppose there is a way to get the track title, at least, without being on channel 1-4 because the Kuvo box requests the data from CDJ mixers, up to all four channels.
Either it's doing that, or Pioneer is storing track IDs. I tend to doubt it, but I haven't sniffed what Rekordbox is sending to Pioneer.
It'd be helpful if there was some way to tear apart the firmware image, but it doesn't seem to match any known formats. It can't be too esoteric though, it contains a lot of plain text strings.
James Elliott
@brunchboy
I have heard that about Kuvo, but I have never had access to a unit, none of the clubs in Madison have them, so I am puzzled about how it works. I hope someone can sniff one and share the details soon! I would also love to know the format of the packets that you use to request wave forms and the beat grid (and the format of the responses). Can you share that? Is the information people have figured out about DBServer organized and published anywhere, beyond what I have done in the dysentery protocol analysis paper?
Austin Wright
@awwright
I suppose I could publish what I've got
Unfortunately by the looks of it this would be a hard protocol to fuzz
In summary though, there's a 2-byte field for (what I presume to be) the method ID, and there's different methods for things like album art, waveform summary x2004 (the 300 px wide version, one byte per pixel), a waveform detail x2904 (300 pixels per second, one byte per pixel), and beat grid info x2204 (uint8 measure_beat, 0x000000, uint32 beat_ms_offset)
Austin Wright
@awwright
Eh, I don't have it documented very well, it's all in code
about 3500 lines of it
I'll figure something out over the weekend, I don't get a lot of time to work on this anymore
I also invented a lot of names for things. Method calls can be fragmented up into multi-message responses, and each response can have multiple fields I called Kibbles
Austin Wright
@awwright
It's also fun to note that all of the menu drawing for the Link functionality is very low-level and done over the network. The CDJ completely re-requests all the menu items every time it has to scroll the menu.
I call the handshake packets Hello and Sup
Austin Wright
@awwright
Austin Wright
@awwright
The other thing to keep in mind about the protocol is it's reverse compatible with every Pro DJ Link product they've ever put out, so there's kludges upon kludges
James Elliott
@brunchboy
I noticed that about the menu drawing when I was looking at network captures a few months ago too, and laughed. Thanks so much for sharing this! Skimming your protocol.txt notes, it looks like we went through incredibly similar research paths. I can see I have a bunch more experimenting ahead of me. I have pulled together my own notes here: https://github.com/brunchboy/dysentery/blob/master/doc/Analysis.pdf
James Elliott
@brunchboy
It’s crazy how months can go by with no activity in my open source projects, and then everything happens at once. I have not had a chance to dig into your insights yet because I just received some breakthrough help that is going to enable a version 1.0 release of CoreMidi4J, which lets MIDI work properly in Java on the Mac. But I am so impatient to do new tricks with my CDJs!
James Elliott
@brunchboy
Just looking at dbstruct.js now that I’ve been able to release CoreMidi4J 1.0, and starting to see the Kibble boundaries you identified, I can see there are going to be some great improvements to the packet analysis PDF! How would you like to be credited? :grin:
moritz bust
@busti
@brunchboy, is it possible to use dysentery from java?
I guess this is probably a clojure related question rather than being related to dysentery.
James Elliott
@brunchboy
I am a little confused by the question, because dysentery is not something that you would use from a program, it is just a lab bench for interactively experimenting with the protocol. Once things get figured out, I implement them in a robust and documented way in beat-link, which is a Java library designed to be convenient to use from Java, Clojure, or any JVM-hosted language. What are you trying to do?
James Elliott
@brunchboy
(To answer your more abstract question, it is possible to write Clojure programs that are designed to be called from Java; you can implement specific interfaces and methods, and subclass objects using Clojure’s java interop features. I do that in afterglow-max which hosts Afterglow as a plugin for the Max visual music/signal processing environment. But Clojure that is not designed to be called from Java is awkward to use from Java, which is the main reason I wrote beat-link in ordinary Java, even though I find that so tedious and slow compared to Clojure these days. And dysentery is not designed to be called by anything.)
James Elliott
@brunchboy
@awwright Wow, there is so much in this code, and it is rough going for me since I am not familiar with Node, and have mostly avoided JavaScript, but I can already tell it is going to make Beat Link so much better! I did not even know about the packets sent to port 50001, which will make sync far more precise, and you have filled in some missing values for me in DJMDevice.playStateMap. (Although I know the meaning of some values that are missing in libdjm.js: 0 means no track is loaded, 4 means a loop is playing, and 17 means the end of the track has been reached and the player has stopped.)
Austin Wright
@awwright
@brunchboy Austin Wright is fine
I would probably choose a different name for Kibble{foo} now, maybe Field or something
But the good thing about nonsense names is it doesn't matter
Austin Wright
@awwright
@brunchboy Also I don't think it's very clear in there, my library has an option to listen for copies of packets sent to it over a TCP channel in a TZSP-like format; it's not something native to any Pioneer device.
To generate the copies of the packets I have 5 USB-Ethernet adapters plugged into a Linksys router running a C program.
That's actually one way I was able to capture data on a 4-CDJ job, except it doesn't capture any data from the device the USB is plugged into (of course)
James Elliott
@brunchboy
It took me a while to figure out what the heck that TZSP stuff was, yes, but I thought that was very clever!
Another byte you may want to look at is 0x89 in the packets the CDJs send to port 50002. I call it F in the protocol analysis, and it is a bit field with interesting status values for the player, including Play, Master, Sync, and On-Air bits.
Austin Wright
@awwright
For the paper it'd probably be useful to describe the packet formats in terms of a C struct, and endian-ness. Perhaps "uint32BE byte[0..3]"
You'll notice that multi-byte numbers area always aligned (like a C struct), 32-bit integers are going to start on byte 0, 4, 8 etc.
James Elliott
@brunchboy
That kind of word alignment is strangely inefficient for a network protocol, I hadn’t noticed that. But alignment doesn’t really make a difference for modern languages, what we really need to know is the offset of the field. I use language-independent byte-field diagrams since I don’t want to make any assumptions about what people are implementing things with. For the next release of the document, though, I think I am going to change the indices in the margins to hex, that is a lot easier to read and do mental arithmetic with! I do mention that the integers are big-endian, and that’s the default network byte order. I’m about to push a couple changes to dysentery that start incorporating some of your discoveries. :sparkles:
Austin Wright
@awwright
Very nice
James Elliott
@brunchboy
Nice indeed, thanks again! I have dysentery properly parsing and constructing all of the type-tagged fields (kibbles) and messages that make up transactions now, and got the track metadata download working from start to finish that way last night. I discovered a new itemTypeLabel value that you may want to add as well: 0x2e means the message contains information about when a track was added to the collection (it seems to propagate from iTunes on my system). The last message before the menu footer in a track data response has an item type with this value, and its label 1 string contains a date in the form yyyy-mm-dd.
James Elliott
@brunchboy
I’ve also discovered that a message type of 0x1004 gives you the track list (sorted by title), whose menu items are of type 0x704. And message type 0x1002 gives you the artist list (sorted by name); those menu items are of type 0x07 which we already knew meant artist.
James Elliott
@brunchboy
All right, @awwright, I seem able to retrieve waveform data, but I don’t know how to display it. Unlike the album art, it is not in a standard image file format. You mention that it is one byte per pixel, but what order are the pixels in? And what do the bytes mean? How do they map to a color? It doesn’t seem like there is enough data there, I am only getting back 900 bytes… Also, more worryingly, once I request waveform data, my next request, whatever it is, gets back a message with type 0x0100, and no arguments, and then the socket closes. Wait, I just checked, and that 0x0100 message is coming right after the 0x4402 message with the 900 bytes of waveform summary even if I don’t send any more messages. So clearly I must be doing something wrong. I guess it is time to bust out my spanned switch and do some more network captures, unless you have a better idea?
James Elliott
@brunchboy
I wonder why there is so much seemingly wasted space in the beat grid, and why it uses little-endian numbers for the times, when the other messages all use big-endian numbers. And why bother storing the 1,2,3,4,1,2,3,4 beat values in there at all when they are so predictable… very strange!
James Elliott
@brunchboy
Also, @Busti, was your question answered? I remain curious what kind of project you are working on. :smile:
moritz bust
@busti
I did not attempt to implement it yet. I'll have access to another cDJ in two weeks though.
The project basically comes down to an Idea fo a graphical user interface I had for controlling lighting.
I should have done Mockup graphics for it. I guess I will do that tommorrow.
James Elliott
@brunchboy
Cool! It’s really fun working on projects like this. I can’t wait to get CDJ waveforms scrolling along the top of my Ableton Push 2 controller which is running my Afterglow light shows, so I can see ahead of time what is coming up in the music.
moritz bust
@busti
I will inform you once I have the graphics done. I think that it might be interesting to you. ☺
Austin Wright
@awwright
@brunchboy For the beat grid, it can start on any beat (1..4), and if you program a tempo change in Rekordbox, it's possible to skip beats or duplicate beats.
For the waveform, I think it's the 3 MSb are color (blue through white), and the 5 LSb are amplitude. There's also several bytes at the start and end that don't seem to be part of the track, I don't know what those mean.
James Elliott
@brunchboy
Ah, ok, that makes sense about why the beat numbers are stored then, but there are still more than twice as many bytes in the structure as there seems to be a need for! As you said a few days ago, there seems to be a lot of accumulated cruft in the protocol. I’ve started updating the analysis document now that I have some working code, I hope to get an updated version pushed this weekend.
James Elliott
@brunchboy
Now for the waveform, it sounds like you are saying it is not one byte per pixel, but rather one byte per segment? So each byte is a color, and the height of the waveform at that point? That makes a lot more sense. I wonder why I was getting 900 bytes for the summary, though, and why it was crashing my player? I need to do another capture to figure out what I got wrong in the request. You don’t happen to have any code that interprets the waveform that is shareable, do you? I couldn’t find anything like that looking through the libpdjl repository. But for now I have plenty to keep me busy documenting what I have already learned, and using the beat grid to generate MIDI timecode.
And thanks, Moritz, I bet it will be interesting to me.
James Elliott
@brunchboy
Hey, @awwright, as I am writing the explanation of the different kind of field types, I just realized that the 0x10 kibble which we thought contained two values with a mysterious 0x0f in between them is probably actually two kibbles! 0x10 introduces a two-byte big-endian integer (the message type), and 0x0f introduces a one-byte integer (the argument count).
Austin Wright
@awwright
@brunchboy I'm not sure what you mean by segment? More specifically, the waveforms use something like:
let value = value of next byte in waveform data
let colorRGB = ( value & 0b11100000, byte & 0b11100000 , 0xff );
let height = value & 0b00011111;
there's one byte per column of pixels to be drawn.
The waveform summary will be 300px, the entire waveform will be ~150 bytes (and pixels) per second of track.
James Elliott
@brunchboy
Right, so when you say pixels there you mean columns of pixels. That’s what I was trying to get at with “segments”. :smile:
James Elliott
@brunchboy
Anyway, thanks for clarifying that. I will write some code to render bytes that way, and compare them to what I see on the CDJs, and hopefully that will help me figure out how far into the response the actual waveform data begins. First I am going to run another capture so I can see the details of the request, and figure out what I was doing wrong.