Noise reduction with sampling software

[TrickyNekro]

Hello guys,

 this time I have problems with my ADC reading from a current sensor...
It seams that any device that generate a amount of noise ( DC motors and such)
affects badly the readings on the ADC... Note that I use the Atmega8 for my project...
Since I have tried anything on electronics level to reduce the noise, and the problem persists....
And because I cannot fill my "device" with filtering capacitors cause every device connected has a different
frequency noise generation on the supply line... I thinking of an algorithm that collects some
values, gets an average, and then, cause if you have studied basic statics you will know that average is
effected badly by extreme boarder values, try to see how many values over time are in the same area,
so based on a factor, the machine will try to use the value which is more shown and closest to the average....
I know it's a little bit difficult, but I did statistics and array programming last year so I can get something by...

How to see my idea... What other, easily implemented ways of filtering I have???
This is for my tutorial, at the members page... So the better the code, the better for all of us!!!

Best Regards, Lefteris
Greece

[superchiku]

if u have ever noticed then ull know that the noice induced has a specific range of values... may be certain wavelength and all ...soooo if u use a proper threshold and try elmiination anything above or below or from a certain range then i guess u can do some kinda filtering...but then again this technic is not completely foolproof if u have fluctuating value of noise.. .

[TrickyNekro]

Since the device I'm building is a bypass device, measuring voltage and current draw, this is not possible...
I think I told that above
What I need is active filtering, the noise frequency and amplitude WILL vary....

[superchiku]

do u know that if u want to do bypass filtering then why dont u use a fbypassing filter instead ...finding a solution in soft will never be accurate..

[cosminprund]

So you want to take an average of the sampled data but you want to remove the top N% and bottom N% samples first because that's where most of the noise is. To do that you'll need to record all readings plus reading frequency and when you're done sampling remove the possibly bad samples from the pack.

Here's how to do it:
You set up an array that records every possible reading value and each cell in that array records the number of occurrences for that value. After you're done sampling you go over the array from top to bottom and then from bottom to top and remove (decrement) the "N%" samples. You then calculate an weighted mean of the remaining values and that's it!

Taking this step by step, the array that holds the number of occurrences needs to hold values big enough to count your total number of samples and have enough cells to count every possible ADC value. If you use 8bit ADC then your array will need to be 256 wide. If you plan to average a maximum of 256 values then each value in the array will be an char; If you want to average more then 256 values then each value will be an integer. It all boils down to what precision you need and how much SRAM you can afford for the job.

Array for 8bit ADC and maximum 256 readings, only uses 256 bytes of SRAM:

Code: [Select]

unsigned char CountValues[256];

Array for 10bit ADC and maximum 256 readings, uses 1Kb of SRAM:

Code: [Select]

unsigned char CountValues[1024];

Array for 10bit ADC and maximum of 65000 readings, uses 2Kb of SRAM (assuming "int" is an 16 bit integer, and I think that's how it is on an MCU):

Code: [Select]

unsigned int CountValues[1024];

After you decide on the kind of array you'd be using you'll need to sample your ADC and fill in the "CountValues" array. Let's assume you'll be taking 100 readings and they'll be 8bit readings. Here's how you fill the array:

Code: [Select]

  unsigned char CountValues[256];
  // You first reset the counter values
  for (int i=0;i<256;i++) CountValues[i] = 0;
  // Now fill in 100 ADC readings
  for (int i=0;i<100;i++)
  {
    unsigned char AdcReading = GetAdcReadingFromCurrentSensor();
    CountValues[AdcReading] += 1;
  }

// After you run this code you'd have something like this in the array:

Code: [Select]

  CountValues[0]==2;
  CountValues[1]==4;
  CountValues[2]==4;
  ....
  CountValues[100]=10;
  ...
  CountValues[253]=5;
  CountValues[254]=3;
  CountValues[255]=2;

// Let's say you want to ignore the lowest 5 values and the highest 5 values (before taking the average). Here's an algorithm that will do that:

Code: [Select]

  unsigned char CountValues[256]; // asuume this is filled in.
  unsigned char ToBeRemoved;
  int i;
  // Start removing values from the top
  ToBeRemoved = 5;
  i = 255;
  while (ToBeRemoved) {
    if (CountValues[i])
      {
        // My cursor is on value "i"; It's count is NOT zero so there's something to be removed!
        CountValues[i] -= 1; // Remove the value from the array
        ToBeRemoved -= 1; // Mark the value removed
      };
    else
      {
        // My cursor is on value "i"; While doing ADC I've seen no [i] value so the counter is 0; Or i've
        // removed all values! Move on to the next value
        i -= 1;
      }
  }

This algorithm is guaranteed to finish: Since I've put into the array 100 values and I'm removing 10, I'll surely find at least 10 values so there's no problem. The algorithm for removing from the bottom is the same, except I'll initially set "i=0" (start from bottom) and increase the cursor in the "else" branch (to move to higher values).

Here's how the array should look like after running this algorithm:

Code: [Select]

  CountValues[0]==0;
  CountValues[1]==1;
  CountValues[2]==4;
  ....
  CountValues[100]=10;
  ...
  CountValues[253]=5;
  CountValues[254]=0;
  CountValues[255]=0;

Now all you need to do is take the weighted average:

Code: [Select]

  int Summ = 0;
  for (int i=0;i<256;i++) // You need to go over the whole array!
    Summ += i * CountValues[i];
  Average = Summ / 90; // Fixed value; It's the number of sampled values (from your "sample" loop) minus the number of elements you removed.

Please note the code is "PSEUDO" code - ie: I wrote it in the forum window. It will probably not compile, but should give you a good idea of what I'm thinking of.
Hope it helps.

[TrickyNekro]

well, that was a hell of an answer and thank for that...
I need some time to read all that stuff and see what you mean...
and yup that was a good method...

The only thing I want to add is that (I don't know if I missed that somewhere....)
to de-selecting the extreme values seems to be quit easier.... You don't have to remove them by writing zero...
You can just not include them...
You can issue a bubble code to the array and have the results in the array in an order that is easily
converted and can extract data with easy....

When you have the data in an order (for smaller value to bigger or vis versa) you can reject extreme values...
To make code even smarter I have though of this...

You can get the average without rejecting any values, then get the average again rejecting the some extreme values, if the difference between the two averages is greater than a want level then do this process again...
if not get the average of the two average and continue....

I have tested the 50 10bit values average method without any other process so far... Although I get very little noise, I want no noise,
The truth is that this project isn't only for a tutorial I write for the forum but also
I'm gonna present it to university... the better it is the better opinion they gonna have about me.....

Thanks for the help so far, and I want to turn this topic to a possible suggestion topic and not a
I NEED HELP!!!!!!! NOOOBBBBB!!!!! topic.........

So, if possible, write just anything you want any know about noise filtering in software...

Thanks in advance,
Regards, Lefteris
GReece

[cosminprund]

I think you missunderstood my algorithm. I'm not storing the values I'm receiving but how many times I received each and every basic value. This is quick and easy because I allready know the ADC only returns 1024 distinct values (from 0 to 1023) in 10bit mode or 256 distinct values (from 0 to 255) in 8 bit mode.

On the other hand you want to store the actual values.

Your method uses less SRAM if you want to average an small number of samples (and also reject extreme samples). Your method uses way more CPU time and space - but that's most likely less of a concern, considering your target Atmega8 only has 1Kb of SRAM available!

My method uses an fixed amount of SRAM to provide 1-256 samples or 257-65535 samples. It's time efficient because it doesn't do any sorting. It's space efficient because the algorithm is very short and simple. Is bad on the Atmega8, especially if want 10bit samples because you don't have enough SRAM!

If you want to take this an step further and present it at the university make sure you do some reading on statistical data analyses. I don't think you can apply the usual noise reduction algorithms because you have absolutely no information on how the signal really needs to look like so no way of smoothing it out. If you find some math formula to apply to the data and you need help turning that into an algorithm just post a link

[TrickyNekro]

The good thing about the signal is that it is lineal or almost lineal....
I'm trying to measure the current through a resistor using an op amp and such...
So the more the current the bigger the value...
Thing is that the current through the resistor won't be a steady value...
This is because it's targeting a motor control system... (basically and ESC)
where electronics motors and other staff exist together....
That darn things generate a hell of noise on the current draw, cause they use PWM to
operate...
These days I was testing the current draw of a DC computer case fan, only to find out that it produces a
500Hz noise on the Op amp output!!!! DARN IT!!!! I also used a high current ESC.... same thing different
range.... The switching of the supply of a brushed and brushless motor did the trick and my life a hell!!!!
When I was stalling the motors, no problem... when they were running.... a pile of......................

So I gotta filter the high frequency noise and in order words make a low pass filter... with software...
So I try to measure for a specified time period the signal as many times more possible and
then do the calculation...
I'm not concert, as you rightly said about Flash but SRAM... with 50 10bit samples I'm
already using 100bytes of data let alone other bytes used as program pointers etc etc etc....
So up to now I assume that I use 150Bytes for the program (I'm not home to tell you exactly how many)

I have more SRAM to use I know, but this is only the starting code...

EEeggghhhh..... what else........ well....

I also try to keep the samples as low as possible, as you said it's a time consuming process...
Although the microcontroller is only sampling two ADC channels and one with hopefully no noise on it...


Darn thing that I'm building... only generating a problem after another.... I recently changed the Op amp to get one with rail-rail output... only to find out that this darn thing is leaking power to the circuit somehow....
aaaa......... I'll be very happy when I'm done with this for sure............

Thanks for the interest and the help so far...
I'll try to keep myself calm about it...

Best of regards, Lefteris
Greece


BTW, I'm first grade at university, but electronics are fifth to sixth grade!!!
I was told to make a presentation of what I have built so far.... but I really
don't want to show that plain robots, I have built so far... thus I'm making a
"research" on a current monitoring circuit I have built myself!!!!!
I guess I'll post pictures of the presentation, if I have at the tutorial page!!!