Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 03 20:26

    brunchboy on master

    Turn off broken "Edit this Page… (compare)

  • Oct 01 22:12

    brunchboy on master

    Add antora and netlify credits … (compare)

  • Sep 08 16:14

    brunchboy on master

    Update other dependencies. (compare)

  • Sep 08 16:04

    brunchboy on master

    Update CoreMidi4J. (compare)

  • Jun 11 03:19

    brunchboy on master

    Update to link to netlify hoste… (compare)

  • Jun 11 03:01

    brunchboy on master

    Update doc readme to reflect us… (compare)

  • Jun 10 21:40

    brunchboy on master

    Fix typo in function name. (compare)

  • Jun 10 21:18

    brunchboy on master

    Remove insecure HTTP link, than… Try simplifying API doc path, u… (compare)

  • Jun 10 20:48

    brunchboy on master

    Working on netlify doc build. (compare)

  • Jun 10 20:27

    brunchboy on master

    Update Clojure version. Update Codox version. Try setting up Netlify build fo… (compare)

  • May 27 18:14
    brunchboy edited #70
  • May 27 18:12
    brunchboy edited #70
  • May 27 18:12
    brunchboy labeled #70
  • May 27 18:12
    brunchboy opened #70
  • May 22 05:21
    brunchboy closed #69
  • May 22 05:21

    brunchboy on master

    Update jquery.mincolors color p… Start wedding reception show. (compare)

  • May 22 01:44
    brunchboy opened #69
  • May 22 01:42
    brunchboy closed #23
  • May 22 01:41
    brunchboy closed #65
  • Apr 29 21:08

    brunchboy on master

    Fix more broken images due to w… (compare)

Austin Wright
@awwright
Hey, I just stumbled across this and it looks like I'm doing something extremely similar except in Node.js; I've got a networkable lighting controller I've been designing since 2015 that does timecode sync to PDJL, layer-based effects, and support for MIDI controllers and Art-Net output. I've used it for several events of 50-200 people.
I can watch the CDJ waveform from the lighting control and see ahead of time when the drop really hits, etc.
Hardware is a Linksys router, 2x CDJ-2000NXS, DJM-2000NXS, 2x Lenovo X220 tablet, Behringer X-Touch, Launchpad Mini, an Art-Net to DMX box, and a few other MIDI devices and Ethernet switches.
James Elliott
@brunchboy
That does sound very cool and like we are exploring a lot of similar ideas! I’d love to know more about how you get at the waveforms. Is your source code and documentation on GitHub?
Austin Wright
@awwright
@brunchboy I have it on a private BitBucket . It looks like the DBServer protocol has been figured out by some other people more or less, it's a simple RPC protocol over TCP with typed arguments and responses
Getting the waveform is one such call only available to clients registered on CDJ channels 1-4
You ask for the waveform of a track ID and it gives back a series of bytes, 5 bits for amplitude and 3 bits for color
Another call lets you get the beat grid, with measure beat number and ms offset, which you can use to figure out where in the track the CDJs are playing
It's the only way available to get the position, and it's what the CDJs use internally when you double the playing track from the Master onto another player using the Sync button
Austin Wright
@awwright
I suspect there's an alternate way to get at the metadata that doesn't require you be on channel 1-4; Armin van Buuren travels with a Macbook Air that lets his LD and Resolume techs see what he's playing, and it works on all four channels
It's probably the case that the beatgrid and waveform is pre-downloaded onto that laptop
James Elliott
@brunchboy
Yes, that is what I have always assumed in my own metadata research, that people will need to pre-analyze tracks and cache all the metadata they are going to want to use during a performance if they can’t use a player number from 1 to 4. I have figured out how to get the track text metadata, but not yet the wave form or beat grid. The beat grid especially would be incredibly useful for my users who want timecode. If you ever decide you are willing to contribute this information to the open source world, please let me know, I would love to update the protocol analysis document and my own reference implementation!
The place where I have gone furthest in working with CDJs is in beat-link-trigger which some producers and artists have been using to sync video walls with CDJ tracks in Resolume, and also to sync Ableton Live improv performances with CDJs using the Ableton Link protocol at music festivals.
Austin Wright
@awwright
Also I suppose there is a way to get the track title, at least, without being on channel 1-4 because the Kuvo box requests the data from CDJ mixers, up to all four channels.
Either it's doing that, or Pioneer is storing track IDs. I tend to doubt it, but I haven't sniffed what Rekordbox is sending to Pioneer.
It'd be helpful if there was some way to tear apart the firmware image, but it doesn't seem to match any known formats. It can't be too esoteric though, it contains a lot of plain text strings.
James Elliott
@brunchboy
I have heard that about Kuvo, but I have never had access to a unit, none of the clubs in Madison have them, so I am puzzled about how it works. I hope someone can sniff one and share the details soon! I would also love to know the format of the packets that you use to request wave forms and the beat grid (and the format of the responses). Can you share that? Is the information people have figured out about DBServer organized and published anywhere, beyond what I have done in the dysentery protocol analysis paper?
Austin Wright
@awwright
I suppose I could publish what I've got
Unfortunately by the looks of it this would be a hard protocol to fuzz
In summary though, there's a 2-byte field for (what I presume to be) the method ID, and there's different methods for things like album art, waveform summary x2004 (the 300 px wide version, one byte per pixel), a waveform detail x2904 (300 pixels per second, one byte per pixel), and beat grid info x2204 (uint8 measure_beat, 0x000000, uint32 beat_ms_offset)
Austin Wright
@awwright
Eh, I don't have it documented very well, it's all in code
about 3500 lines of it
I'll figure something out over the weekend, I don't get a lot of time to work on this anymore
I also invented a lot of names for things. Method calls can be fragmented up into multi-message responses, and each response can have multiple fields I called Kibbles
Austin Wright
@awwright
It's also fun to note that all of the menu drawing for the Link functionality is very low-level and done over the network. The CDJ completely re-requests all the menu items every time it has to scroll the menu.
I call the handshake packets Hello and Sup
Austin Wright
@awwright
Austin Wright
@awwright
The other thing to keep in mind about the protocol is it's reverse compatible with every Pro DJ Link product they've ever put out, so there's kludges upon kludges
James Elliott
@brunchboy
I noticed that about the menu drawing when I was looking at network captures a few months ago too, and laughed. Thanks so much for sharing this! Skimming your protocol.txt notes, it looks like we went through incredibly similar research paths. I can see I have a bunch more experimenting ahead of me. I have pulled together my own notes here: https://github.com/brunchboy/dysentery/blob/master/doc/Analysis.pdf
James Elliott
@brunchboy
It’s crazy how months can go by with no activity in my open source projects, and then everything happens at once. I have not had a chance to dig into your insights yet because I just received some breakthrough help that is going to enable a version 1.0 release of CoreMidi4J, which lets MIDI work properly in Java on the Mac. But I am so impatient to do new tricks with my CDJs!
James Elliott
@brunchboy
Just looking at dbstruct.js now that I’ve been able to release CoreMidi4J 1.0, and starting to see the Kibble boundaries you identified, I can see there are going to be some great improvements to the packet analysis PDF! How would you like to be credited? :grin:
moritz bust
@busti
@brunchboy, is it possible to use dysentery from java?
I guess this is probably a clojure related question rather than being related to dysentery.
James Elliott
@brunchboy
I am a little confused by the question, because dysentery is not something that you would use from a program, it is just a lab bench for interactively experimenting with the protocol. Once things get figured out, I implement them in a robust and documented way in beat-link, which is a Java library designed to be convenient to use from Java, Clojure, or any JVM-hosted language. What are you trying to do?
James Elliott
@brunchboy
(To answer your more abstract question, it is possible to write Clojure programs that are designed to be called from Java; you can implement specific interfaces and methods, and subclass objects using Clojure’s java interop features. I do that in afterglow-max which hosts Afterglow as a plugin for the Max visual music/signal processing environment. But Clojure that is not designed to be called from Java is awkward to use from Java, which is the main reason I wrote beat-link in ordinary Java, even though I find that so tedious and slow compared to Clojure these days. And dysentery is not designed to be called by anything.)
James Elliott
@brunchboy
@awwright Wow, there is so much in this code, and it is rough going for me since I am not familiar with Node, and have mostly avoided JavaScript, but I can already tell it is going to make Beat Link so much better! I did not even know about the packets sent to port 50001, which will make sync far more precise, and you have filled in some missing values for me in DJMDevice.playStateMap. (Although I know the meaning of some values that are missing in libdjm.js: 0 means no track is loaded, 4 means a loop is playing, and 17 means the end of the track has been reached and the player has stopped.)
Austin Wright
@awwright
@brunchboy Austin Wright is fine
I would probably choose a different name for Kibble{foo} now, maybe Field or something
But the good thing about nonsense names is it doesn't matter
Austin Wright
@awwright
@brunchboy Also I don't think it's very clear in there, my library has an option to listen for copies of packets sent to it over a TCP channel in a TZSP-like format; it's not something native to any Pioneer device.
To generate the copies of the packets I have 5 USB-Ethernet adapters plugged into a Linksys router running a C program.
That's actually one way I was able to capture data on a 4-CDJ job, except it doesn't capture any data from the device the USB is plugged into (of course)
James Elliott
@brunchboy
It took me a while to figure out what the heck that TZSP stuff was, yes, but I thought that was very clever!
Another byte you may want to look at is 0x89 in the packets the CDJs send to port 50002. I call it F in the protocol analysis, and it is a bit field with interesting status values for the player, including Play, Master, Sync, and On-Air bits.
Austin Wright
@awwright
For the paper it'd probably be useful to describe the packet formats in terms of a C struct, and endian-ness. Perhaps "uint32BE byte[0..3]"
You'll notice that multi-byte numbers area always aligned (like a C struct), 32-bit integers are going to start on byte 0, 4, 8 etc.
James Elliott
@brunchboy
That kind of word alignment is strangely inefficient for a network protocol, I hadn’t noticed that. But alignment doesn’t really make a difference for modern languages, what we really need to know is the offset of the field. I use language-independent byte-field diagrams since I don’t want to make any assumptions about what people are implementing things with. For the next release of the document, though, I think I am going to change the indices in the margins to hex, that is a lot easier to read and do mental arithmetic with! I do mention that the integers are big-endian, and that’s the default network byte order. I’m about to push a couple changes to dysentery that start incorporating some of your discoveries. :sparkles:
Austin Wright
@awwright
Very nice
James Elliott
@brunchboy
Nice indeed, thanks again! I have dysentery properly parsing and constructing all of the type-tagged fields (kibbles) and messages that make up transactions now, and got the track metadata download working from start to finish that way last night. I discovered a new itemTypeLabel value that you may want to add as well: 0x2e means the message contains information about when a track was added to the collection (it seems to propagate from iTunes on my system). The last message before the menu footer in a track data response has an item type with this value, and its label 1 string contains a date in the form yyyy-mm-dd.
James Elliott
@brunchboy
I’ve also discovered that a message type of 0x1004 gives you the track list (sorted by title), whose menu items are of type 0x704. And message type 0x1002 gives you the artist list (sorted by name); those menu items are of type 0x07 which we already knew meant artist.
James Elliott
@brunchboy
All right, @awwright, I seem able to retrieve waveform data, but I don’t know how to display it. Unlike the album art, it is not in a standard image file format. You mention that it is one byte per pixel, but what order are the pixels in? And what do the bytes mean? How do they map to a color? It doesn’t seem like there is enough data there, I am only getting back 900 bytes… Also, more worryingly, once I request waveform data, my next request, whatever it is, gets back a message with type 0x0100, and no arguments, and then the socket closes. Wait, I just checked, and that 0x0100 message is coming right after the 0x4402 message with the 900 bytes of waveform summary even if I don’t send any more messages. So clearly I must be doing something wrong. I guess it is time to bust out my spanned switch and do some more network captures, unless you have a better idea?
James Elliott
@brunchboy
I wonder why there is so much seemingly wasted space in the beat grid, and why it uses little-endian numbers for the times, when the other messages all use big-endian numbers. And why bother storing the 1,2,3,4,1,2,3,4 beat values in there at all when they are so predictable… very strange!
James Elliott
@brunchboy
Also, @Busti, was your question answered? I remain curious what kind of project you are working on. :smile:
moritz bust
@busti
I did not attempt to implement it yet. I'll have access to another cDJ in two weeks though.
The project basically comes down to an Idea fo a graphical user interface I had for controlling lighting.
I should have done Mockup graphics for it. I guess I will do that tommorrow.