Adapt Webbotlib to compile the full library and not use .a files.

[madsci1016]

As per this post,

I'd like to be able to configure WinAVR to compile the full library every time and not use .a files. Is this possible? I don't see why not, since webbot provides all the source; but I don't know how it would be done. How can I go about doing this?

On the plus side - whilst it may be a loss for the US/Navy then it can still be a plus for the rest of the world 

 

Make everyone use Project Designer whereby the pre-compiled code can also be MUCH smaller and may even run on ATtiny chips.

I think i've helped this cause a little, don't you?

[Admin]

Although for other reasons I see it as a good thing, there is a contradiction with your logic here . . .

Of course you don't like running mysterious pre-compiled code for security/reliability reasons, but Project Designer itself is mysterious pre-compiled code that connects to a server transferring who-knows-what every time you run it

My understanding is the code is precompiled using software called Ant, but I was never able to get it to work on my laptop, and I couldn't find instructions helpful enough for when I got stuck.

here are a few relevant excerpts from emails with Webbot:

Are you familiar with Jakarta Ant - thats what I use to compile my libs. ie 'ant release' will rebuild my lib for all platforms, create ZIP files, etc. Rather than a 'make file' it just has a 'build.xml' file to drive it. Its not 100% brilliant for C in its dependency checks so normally I 'clean' then 'rebuild'. Perhaps I should set you up as a 'contributor' on the project page. Then you can edit/rebuild/check-in changes which I can always 'veto' on lib style ;-)

...

There's the benefit of using 'ant'- as command line order is written once in the 'makefile'. Every time I do a release it also recompiles all my 'test' examples - to make sure they compile. So its like a makefile of makefiles. None of this flakey IDE stuff.

...

go here http://ant.apache.org/bindownload.cgi
In the 'Current Release of Ant' section click on the link for .zip archive: apache-ant-1.7.1-bin.zip
Download the zip file and unpack to c:\ as all the files will go to 'c:\apache-ant-1.7.1
Only thing left to do is to add: C:\apache-ant-1.7.1\bin to your PATH - guess you know how to do that.
Test by launching a DOS box and typing: ant -version
You should see the version number. If not then Ant is not on your PATH.
If all is well: then in the DOS box - change to the folder for Webbot library.
Copy the attached build.xml file to that directory
Type: ant -projecthelp
You should see:
Main targets:

ATMega168       ATMega168 library
ATMega2560      ATMega2560 library
ATMega32        ATMega32 library
ATMega640       ATMega640 library
ATMega8         ATMega8 library
build-lib       Make all libraries
clean           clean everything
clean-examples  Clean the examples
clean-lib       Clean the library
examples        Examples
make-example    internal use only
makeone         internal use only
release         Create release
Default target: build-lib

This means that just typing:
ant
will do 'build-lib' as the default - ie 'Make all libraries'.
To just rebuld the ATMega640 library type: ant ATMega640
etc
If in doubt then clean everything by typing: ant clean


Hopefully that all works.

PS Decided to use ANT because it is far more friendly than those C make files.
PPS You commented about debugging. I use and test against -Os to make the code small. This knackers AVRStudio. So if you want a non-optimeised version of the library then edit 'build.xml' and change -Os to -O0 then do a clean and build.

And this is where I got stuck:

> Only thing left to do is to add: C:\apache-ant-1.7.1\bin to your PATH -
> guess you know how to do that.
> Test by launching a DOS box and typing: ant -version
> You should see the version number. If not then Ant is not on your PATH.
I get this:
C:\apache-ant-1.7.1\bin>ant -version
Unable to locate tools.jar. Expected to find it in C:\Program Files\Java\jre6\li
b\tools.jar
Apache Ant version 1.7.1 compiled on June 27 2008


> If all is well: then in the DOS box - change to the folder for Webbot
> library.
> Copy the attached build.xml file to that directory
> Type: ant -projecthelp
I put the xml file in WebbotLib. But I get this:
Z:\code2009\WebbotLib>ant -projecthelp
'ant' is not recognized as an internal or external command,
operable program or batch file.

Good luck!

[madsci1016]

Although for other reasons I see it as a good thing, there is a contradiction with your logic here . . .

Of course you don't like running mysterious pre-compiled code for security/reliability reasons, but Project Designer itself is mysterious pre-compiled code that connects to a server transferring who-knows-what every time you run it

You are right, which is why I was never going to use project designer at work. Did I mention otherwise? 

I also tried to follow instructions from webbot on how to use Ant and equally got stuck.

It just really bothers me that it should be easy to have WinAVR do it all since we have all the source.

Can it be done Webbot?

[Webbot]

Not sure why you two guys got stuck in using Ant to build the .a files from the source code.

Presume you have followed Dunk's tutorial at http://www.societyofrobots.com/member_tutorials/node/382 showing how to do it on Win, Unix and Mac. He doesn't seem to have your problems.

Ant is used to build production releases of about 99.9999999% of ALL java applications - so its not some flakey thing its an industry standard thing - and although it runs under Java it can build non-Java code. Even Americans have managed to use it ! 

"Can I just change WebbotLib to only use .h files" ? - No.
For the same reason that even AVRlib uses C files rather than doing everything via H files.
If your application has multiple C files then it gets even harder to manage.

The benefit of using .a files is that all my code has been built using '-Os' to make it smaller and faster and since I've learned to understand the avr-gcc compiler 'bugs' then its all fine and tested. I think I have only ever had one bug ever reported, long since fixed, caused due to this setting.

But most people choose to use the compiler with optimisation disabled '-O0' - mainly coz they dont want to have to bother with learning the 'avr-gcc' bugs that can be introduced. If WebbotLib could be achieved just using 'h' files then this means that everything would be compiled with larger and slower code - which will have a bad impact on interrupt service routines. Unfortunately avr-gcc has no way to use '#pragma', as done by most C compilers, to change the optimisation settings for any given piece of code - its "all or nothing" - which is bad but I haven't written the compiler!

So if your issue is that the '.a' files haven't been built in America - then learn to use Ant so you can build it yourself.

As for Project Designer then it uses Java jnlp technology. The only 'mysterious transfer' is to compare the version on your pc with the version on the server and downloads the new version if required. This is 'pure Java' ie its all done by Java itself and I do not play any part in that communication - hence it shows my digital certificate. Java is now owned by Oracle - an American company - so ask them what it does as I don't have a clue and don't really care since it works.

Whilst you are at it then why not ask your browser provider (Microsoft, Firefox, etc etc) what information is used in an HTML page load to decide whether the cached version is the same as the latest server version so it can decide whether to download the latest version from the server. That's analogous to what jnlp does. Presumably you are also limited to only being able to access American web pages.

Trust that all the hardware and software drivers installed on your computer are 'made in America'. And that you also only ever run programs at work that are shipped as source code and compiled-by-you.

So in summary: WebbotLib is shipped as precompiled .a files (which you can recompile yourself) and this wont be changing; my Board/Project/Gait Designer tools are pure Java and will not be changed nor ever shipped as source code; and 'yes' I am a 'foreigner' (ie not American) and I have no desire to change that status either.

Live with it - or move on !

[Admin]

hmmmm Webbot I think you took me *waaaay* too seriously on my joke with madsci

I was really just poking fun at him, tin foil hat style and all.

To coworkers of madsci: at some point you just got to trust other people and their code. Webbot has a lot more to gain by building his reputation as an awesome coder, and he isn't just some random guy in China no one has heard of before that works super cheap.


But yea, Webbot is definitely right on the point of optimization.

[madsci1016]

Trust that all the hardware and software drivers installed on your computer are 'made in America'. And that you also only ever run programs at work that are shipped as source code and compiled-by-you.

You seem to have taken offense to this. That was not my intention.

The rules for the DoD is we can only use proprietary software that has had it's source code 'inspected' by the government. Or, open source code where the code is accessible. And yes, there is a division is some secret mountain where all they do is inspect source from propriety vendors, compile it for themselves and run hashes to confirm it matches the machine code the vendor distributes. Then it enters a database of 'approved' software DoD wide; and i get to use it on my computer at work. Even the millions of lines of linux source has to be analyzed for each distro before we can use it. Our Pcs are scanned every night for un-approved software.

That said, the project I am working on is a robot that could possibly turn into a naval acquisition, so I have to follow stringent guidelines.  I will be asked to turn over all my source, including libraries used, and will be expected to show that that source compiles to the same machine code on the robot. If i don't know how to do that myself, i can't show the reviewer.

"Can I just change WebbotLib to only use .h files" ? - No.
For the same reason that even AVRlib uses C files rather than doing everything via H files.

I don't understand, what the reason I can't just compile directly with WinAVR? It uses avr-gcc and i could set the same optimization options in my makefile, no?

I would have thought you would have been more likely to help. Since if i did get it too work and used webbotlib, you could turn around and say your library was used on million dollar DoD robots. How cool would that be?

Instead, I've had to resort to Arduino for the time being. Oddly and thankfully enough, the Arduino IDE is approved for use.

This isn't an "American" thing, it's a source to machine code thing. 

To coworkers of madsci: at some point you just got to trust other people and their code.

That will never happen. The amount of regulation and control they exert on our machines make it almost impossible to do work sometimes. The whole no USB thumb drive is really getting to me.

[Webbot]

You seem to have taken offense to this. That was not my intention.

Nor mine - must be frustrating to work in such an environment. But also hopefully rewarding!

I don't understand, what the reason I can't just compile directly with WinAVR? It uses avr-gcc and i could set the same optimization options in my makefile, no?

Guess you've never built a complex library. Yes its easy to add H files that do minimal things like flipping pins. Thats easy. Your problem comes when you want code re-use (to keep the code small).

For example. Consider a UART. The ATMegas may have 1,2,3,4 or more of them. Each requires a substantial amount of code to cope with Tx, Rx and the interrupt vectors.

Now try to put this into a single H file. How do you support the 4 or so different UARTs? They use different I/O registers and interrupt vectors.
Some try to get round it by having a 'uart1.h', 'uart2.h' etc etc. But thats a bit of a nonsense - as if you want to use a different uart then you need to change all the #includes.

Now lets say your program has a second, third, etc  C file each of which #includes UART3.h

Big problem!!!!

UART3.h can either define macros/inlines or it can define actual code (like a C file). So where do you put the uart3 rx and tx interrupt handlers? If its in the H file and its included by 3 x C files then you now have 3 x multi defined interrupt vectors - so avr-gcc will give you a 'mega - error'.

So H files are possible if you only have 'one' of something - say like I2C, SPI etc etc but falls down once you have "many" like uart, adc, pwm etc etc. Thats why if you look at other libs like AVRlib then you will see that the latter have a 'C' file and not just an 'H'.

I would have thought you would have been more likely to help. Since if i did get it too work and used webbotlib, you could turn around and say your library was used on million dollar DoD robots. How cool would that be?

Its not a question of helping - its a question of your not understanding 

Instead, I've had to resort to Arduino for the time being. Oddly and thankfully enough, the Arduino IDE is approved for use.

How bizarre!  They authorise an IDE which is just a glorified text editor or developer environment. Presumably they then let you integrate code from anywhere in the world !! If the DoD is as 'strict' as you suggest then someone should be fired over the Arduino IDE thing. 

[dunk]

The rules for the DoD is we can only use proprietary software that has had it's source code 'inspected' by the government. Or, open source code where the code is accessible. And yes, there is a division is some secret mountain where all they do is inspect source from propriety vendors, compile it for themselves and run hashes to confirm it matches the machine code the vendor distributes. Then it enters a database of 'approved' software DoD wide; and i get to use it on my computer at work. Even the millions of lines of linux source has to be analyzed for each distro before we can use it. Our Pcs are scanned every night for un-approved software.

but Webbot provides the source he uses to compile WebbotLib.
he provides the Ant build file as well.
his pre-compiled .a file is just there so you don't have to run Ant yourself.

just give the WebbotLib zip file to your guys in some secret mountain. everything they need is in the .zip.
i'm sure they have *lots* of experience building from source code.

if you want to build the .a file yourself you can find instructions at http://www.societyofrobots.com/member_tutorials/node/382 (as Webbot pointed out).
it's not hard.
Ant is not doing anything magic. you can in theory build the .a file from the command line by invoking gcc your self (many times with many different options) but Ant automates this procedure.


dunk.

[madsci1016]

I think I'm just going to have to write this off as I don't have the required understanding to get this. I still don't see how gcc through ant would be different then gcc through WinAVR.

How bizarre!  They authorise an IDE which is just a glorified text editor or developer environment. Presumably they then let you integrate code from anywhere in the world !! If the DoD is as 'strict' as you suggest then someone should be fired over the Arduino IDE thing.

I think you answered this yourself, it's just a glorified text editor. Very easy to prove it does nothing but edit text.

just give the WebbotLib zip file to your guys in some secret mountain. everything they need is in the .zip.
i'm sure they have *lots* of experience building from source code.

Our project would have to 'pay' the guys there to do the stuff themselves since this is an oddball thing.

[Webbot]

I think I'm just going to have to write this off as I don't have the required understanding to get this. I still don't see how gcc through ant would be different then gcc through WinAVR.

Ant just lets you create a much simpler 'make file' using a different syntax. The one I ship with the library allows you to build the library for 12 different processors. Doing this in a traditional makefile would be a nightmare.

The end result, the .a files, are the same either way.

[dunk]

hey Madsci,
if i understand you correctly, you don't need to understand how Ant works, you only need to be able to compile the code yourself right?
so just download the WebbotLib.zip, unpack it, delete the WebbotLib.a file, run Ant as described in the doc to rebuild the WebbotLib.a file.
job done.


dunk.

[Webbot]

So the scary sounding part here is run Ant as described in the doc to rebuild the WebbotLib.a file

But its as simple as:
1. Launch a DOS/Command Window
2. Change to the root folder of WebbotLib
3. Type in "ant" and press return

It then builds all of the '.a' files from the source code.

[Admin]

Fortunately I get to use USB sticks at our Navy base . . . probably because its a research facility without access to live Navy/government systems. That said, it may be worth it to Webbot and Madsci to get WebbotLib approved for DoD use. I however don't need software approved as code doesn't leave the lab and one-off prototypes.

From a more motivating prospective, Webbot can potentially charge a licensing fee for anything acquired by the DoD that uses WebbotLib.

[madsci1016]

Ant just lets you create a much simpler 'make file' using a different syntax. The one I ship with the library allows you to build the library for 12 different processors. Doing this in a traditional makefile would be a nightmare.

This is the answer I was looking for, and now it makes sense. I'll have to see if I can either figure out Ant (even though the first time was a failure) or try to reverse engineer my own makefile.

Thanks guys.

That said, it may be worth it to Webbot and Madsci to get WebbotLib approved for DoD use.

Like I said earlier, that would cost out program big $$ to do.

Also, i'm envious you get USB drives, and avoid all these headaches.

[Admin]

That said, it may be worth it to Webbot and Madsci to get WebbotLib approved for DoD use.

Like I said earlier, that would cost out program big $$ to do.

Also, i'm envious you get USB drives, and avoid all these headaches.

I'm sure its only like $10k, right? I'm not being sarcastic, this really is pocket change if the Navy gives it a 'go ahead' for field use. I'm sure other groups may also be interested, so costs can be split?

Anyway, I'm getting off-topic here . . .