July 11, 2011

ryanmcgee ryanmcgee
Lab Rat
3 posts

Real-time Audio Processing with Qt

 

Hello, I am experienced with programming C/C++ sound synthesis and effects using PortAudio, RT Audio, and JUCE libraries. The Qt audio input and output code examples have shown me how to access sample buffers for audio input and output, but the methodology is quite different in Qt so I can not figure out how to simply route the incoming audio buffer straight to the output buffer i.e. a simple pass through that would allow me to then add real-time effects along the chain. Other libraries provide a callback that gives pointers to both input and output buffers at block rate, so achieving this is as simple as *out++ = *in++.

In Qt I have separate subclasses of QIODevice for my input and output. The output works great with a simple wavetable synthesizer that I have coded. The input is reading audio correctly and I see the value of the samples increase when tapping on the mic etc. But how to tie one to the other in sync? I gave my output a pointer to input’s sample buffer and I do get audio from the mic but it is choppy and clips often. I set the buffersize of my input and output devices to match, but no luck. I’m assuming there has to be a better way to do this. I do not fully understand QIODevice methodology. I have read through the documentation and am guessing I need to use the readyRead() signal with a custom slot? Does anyone have experience with this or a simple input to output code example?

Thanks!
Ryan

5 replies

July 21, 2011

vidar vidar
Lab Rat
60 posts

The easiest way is to use one subclass of QIODevice where you override the writeData and readData methods. readData could do the processing of the incoming data and store the result in an “intermediate” buffer and writeData uses memcpy to copy this buffer to the *data buffer.

You can initiate the realtime processing by

  1. myProcessing->start(); // subclass of QIODevice
  2.    audioIn->start(myProcessing);
  3.    audioOut->start(myProcessing);

July 21, 2011

ryanmcgee ryanmcgee
Lab Rat
3 posts

That is what I originally tried, but it is not that simple.

So here are the read/writeData methods in my subclass of QIODevice:
(myInputData is the “intermediate buffer”)

qint64 AudioMixer::readData(char *data, qint64 len){

memcpy(data, myInputData, len); return len;

}

qint64 AudioMixer::writeData(const char *data, qint64 len){

memcpy(myInputData, data, len); return len;

}

This does pass the incoming audio to the output as I was hoping to achieve, but it is very choppy. Can you suggest a solution to properly buffer everything? I tried a ring buffer so that the reading is delayed from the writing, but I get the same choppy audio.

Thanks!

July 21, 2011

ryanmcgee ryanmcgee
Lab Rat
3 posts

Strange, also noticed that even though I’m calling:

m_audioOutput->setBufferSize(4096); m_audioInput->setBufferSize(4096);

readData and writeData are not reading/writing the same lengths.

Maybe the solutions involves another way to synchronize the two?

July 22, 2011

Scorx Ion Scorx Ion
Lab Rat
24 posts

I am also working on real-time audio processing using 3rd party libraries and Qt, and have run into a snag. Perhaps if I share what I have done I will help you, and then you or someone else can point out what I am doing wrong and help us both.

First, note that I used the Spectrum Analyser discussed here [labs.qt.nokia.com] and found on your computer at InstallPath\Qt\Demos\4.7\spectrum to come up with most of what I know.

I am using QAudioInput and QAudioOutput for recording and playback. Much like you, I have setup a circular buffer to store the recorded audio (I use a QByteArray as the data structure for my buffer). In theory, as each element in the circular buffer fills up, I can pass it to my audio processing (in my case CMU Sphinx for voice recognition) and then clear that buffer element. For debug purposes, I do not clear the buffer and can choose to play or process any of the individual buffer elements on demand to make sure I have recorded what I expect and can double check the processed output for each buffer. This results in perfect audio recording and playback, however I get inconsistent results from processing the buffer that eventually lead to seg faults after enough processing attempts.

A summary of the code I use is as follows:

  1. // Setup QAudioInput and QAudioOutput, not shown for brevity. See the Spectrum example for details
  2. QAudioInput input;
  3. QAudioOutput output;
  4. QIODevice *intermediateDevice; // This very temporarily stores recorded audio and then transfers it to the circular buffer
  5.  
  6. // Initialize the circular buffer, also shown in more detail in the Spectrum example
  7. QByteArray byteArray[NUM_CIRCLE_BUFFER_ELEMENTS];  // Note that the spectrum example names this variable buffer despite it not being a QBuffer
  8. qint64 byteArrayUsed[NUM_CIRCLE_BUFFER_ELEMENTS]; // This tracks how much of each buffer has been used
  9.  
  10. // Start recording audio
  11. intermediateDevice = input->start();
  12. connect(intermediateDevice, SIGNAL(readyRead()), this, SLOT(captureData())); // captureData is a function you define, and example is below
  13.  
  14. // Example captureData
  15. void captureData(qint64 bytesToCapture)
  16. {
  17.   qint64 bytesAvailable = byteArray[current].size() - byteArrayUsed[current];
  18.   qint64 bytesToRead = qMin(bytesToCapture, bytesAvailable);
  19.  
  20.   // Transfer the audio from the intermediateDevice to the buffer
  21.   bytesActuallyRead = intermediateDevice->read(byteArray[current].data() + byteArrayUsed[current], bytesToRead);
  22.  
  23.   byteArrayUsed[current] += bytesActuallyRead;
  24.  
  25.   // Note that this is the simplified version, you would want to have the transferring in a loop and do this again in the next byteArray if bytesActuallyRead < bytesToRead
  26. }
  27.  
  28. // How to play one of the byte arrays
  29. QIODevice player;
  30. player.close();
  31. player.setBuffer(&(byteArray[current]));
  32. player.open(QIODevice::ReadOnly);
  33. output.play(&player;);

The last piece is handing each of the QByteArrays to your processing libraries. I’ve tried several approaches but haven’t had any consistent results.

Let me know if this is helpful and if you can figure out how to do this last part.

July 22, 2011

vidar vidar
Lab Rat
60 posts

Ryan: you have to take care of the “len” – parameters, they can change on every call. you could implement the buffer as some sort of circular buffer and check your write and read positions …
this should work:

  1. qint64 myProcessing::writeData(const char *data, qint64 len) {
  2.  
  3.  qint16* aVal;
  4.  
  5.  qint64 numSamples = len / 2;
  6.  qint64 posSamples = 1;
  7.  
  8.  aVal = (qint16*) data;
  9.  
  10.  while (posSamples <= numSamples) {  
  11.  
  12.   mBuffer[mBufWritePos] = doSomething(*aVal);
  13.  
  14.   mBufWritePos = (mBufWritePos+1) % bufSize;
  15.  
  16.   aVal++;
  17.   posSamples++;
  18.  
  19.  }
  20.  
  21.  mBufFill += len;
  22.  
  23.  return len;
  24. }
  25.  
  26. qint64 myProcessing::readData(char *data, qint64 maxlen)
  27. {
  28.  if (mBufFill<=0) return 0;
  29.  
  30.  qint64 toDo = qMin(maxlen, (qint64) mBufFill);
  31.  
  32.  if (toDo / 2 + mBufReadPos > bufSize) {
  33.  
  34.   qint64 firstNum = (bufSize - mBufReadPos) * 2;
  35.  
  36.   memcpy( data,
  37.     &mBuffer;[mBufReadPos],
  38.     firstNum );
  39.   memcpy( data + firstNum,
  40.     &mBuffer;[0],
  41.     toDo - firstNum);
  42.  
  43.  } else {
  44.   memcpy(data, &mBuffer;[mBufReadPos], toDo);
  45.  
  46.  }
  47.  
  48.     mBufReadPos =  (mBufReadPos + toDo/2) % bufSize;
  49.     mBufFill -= toDo;
  50.  
  51.     return toDo;
  52.  
  53. }

 
  ‹‹ Phonon snapshot problems      [SOLVED] QNetworkAccessManager does not emit finished() signal ››

You must log in to post a reply. Not a member yet? Register here!