Discussion:
Reflecta ProScan / Crystalscan 7200 PIE film scanner update
(too old to reply)
Michael Rickmann
2012-08-21 20:27:06 UTC
Permalink
It has become a bit silent about these scanners since Jan Vleeshouwers'
post 8 months ago (
http://lists.alioth.debian.org/pipermail/sane-devel/2011-December/029337.html
). Finally, I had some time to collect what else Jan and I had found out
last year and to put it into a patch against the pie backend. Both
scanners identify as PIE SF Scanners. They share the same USB id
0x05e3:0x0145 use a SCSI over USB protocol and can be distinguished by
byte 0x74 in the INQUIRY return block. Support for the ProScan is quite
reasonable now, for the Crystalscan presumably less because calibration
(see below) was mostly guessed from what Jan had snooped and
communicated. I tested with scanimage and XSane. Currently, Klaus Kaempf
seems to work on a PIE MF Scanner (
http://lists.alioth.debian.org/pipermail/sane-devel/2012-July/030092.html )
which is more advanced but presumably not so much different in protocol.
What can the backend do with the SF scanners hardware?
1) scan at resolutions from 20 to 3600 dpi, increment 1
2) read RGB and single pass RGBI data
3) read at 8 and 16 bit [recommended] colour depth
4) calibrate and read in normal and quality [recommended] mode
This is what could be detected in snoops using Cyberview and Silverfast
under Windows. Silverfast does not support normal mode. Calibration and
shading correction is done with the help of software.
What can the backend do by processing the data read?
1) option: clean the infrared colour plane from red spectral overlay
2) option: use the infrared data to remove dirt, output RGB
3) option: simple film grain smoothening (just a triangular blur)
4) option: crop using sanei_magic except for cleaned scans of positive
film where an internal routine detects faster and more correctly
5) options: simple colour conversions, constant gamma, invert
Scanned positive film really looks good, negative film bad due to still
missing colour profiles. Also the options of Xsane for negative film do
not improve things. Scanning a 36x24 mm slide at 3600 dpi takes about 88
sec in RGB, 102 sec in RGBI with infrared discarded, 110 sec in RGBI
with cleaning and cropping. If one compares these numbers with the ones
at http://www.filmscanner.info/en/ReflectaProScan7200.html it seems that
the backend handles the scanner correctly.

What can the backend do in spite of the USB additions? Recently I got
hold of a 12 year old Adlib JetScan 630 SCSI flatbed scanner which is
supported by the pie backend. The scanner still scans at 600 dpi (its
maximum) at “nomal” and “fine” speed; colour, grayscale, halftone and
lineart are ok. However at high resolutions and “pro” speed I get
hardware “integration time adjustment failure”s “too dark” or “too
light”. I get the same errors with SANE-Backends-1.0.22 as included in
Ubuntu 12.04 64-bit. I guess, it is the age or a limitation of this kind
of scanner. So I think that the direct SCSI part of the backend has not
deteriorated.

Description of the added code: I have added three files
pie_usb.h contains basic definitions for the USB code
sanei_ir.h, sanei_ir.c contain code to remove dirt with help of the
infrared plane. I think that this code can migrate to sanei_magic
because it may become useful also for other backends. There is still a
license problem. I have adapted three threshold routines from M. Emre
Celebi's Fourier_0.8 project (
http://sourceforge.net/projects/fourier-ipal/ ) which is licensed under
the GNU General Public License Version 2.
Changed files:
sanei_magic.c : sanei_magic_crop can crop images with >8 bit colour
depth now
pie_scsidefs.h : mainly added support for PIE vendor specific SCSI commands.
pie.c : really a lot. All pure USB functions are prefixed with pie_usb_,
all other functions with pie_ (renamed a few) or sane_. The SCSI over
USB emulation runs via pie_usb_scsi_wrapper which branches into low
level USB code. All other functions talk SCSI. As pie_usb_scsi_wrapper
uses the same arguments as sanei_scsi_cmd functions can use a function
pointer stored in the scanner structure to do direct SCSI or SCSI over
USB. Another possibility to distinguish between the two possibilities is
the “model” variable, also in the scanner structure. If it is NULL it
indicates a SCSI scanner, otherwise it contains a pointer to the
description of an USB scanner. A number of the original functions had
become so messed up with if-thens that I wrote pie_usb_ counterparts to
ensure that the SCSI part of the backend was not suffering.
A note on calibration:
An outline of what happens during a scan you can find in the
introductory comment of pie_usb.h. The best reference for the commands
and what they contain is in Jan's post cited above. The scanners use two
custom SCSI commands for reading and sending calibration records. During
full calibration a record is read from the scanner, then a few
calibration lines are read and last, altered calibration data is sent
back to the scanner. Although Jan and I could figure out the meaning of
most record fields, the snoops did not tell us what was happening
between reading and sending in the Windows software. Reverse engineering
seemed hopeless. I know Jan would like to have an exact calibration
procedure derived from data sheets. There is one huge custom made chip
in PIE scanners which presumably does the essential things, and of
course there is no documentation. So I started a life-science approach
by empirically determining a mathematical model and its parameters. Once
one knows what to send for exposure time, gain and offset, one can try
different values, scan an empty slide and measure mean color values of
the pictures. It melts down to find a relation between exposure time and
gain and resulting brightnesses for R, G, B because the Windows programs
never changed offset or infrared values between calibration reading and
writing. In OpenOffice Calc one can fit a number of curves to the time,
gain an illumination values to get a first idea or one can compare the
measured values to ones of an own function. In the end the functions
giving the best fit were highly non-linear and their parameters had to
be determined with help of OpenOffice's evolutionary solver. For a short
description of how calibration is done look at the comments before the
pie_usb_cali functions in pie.c. If that is not sufficient look at the
pie-usb-calibration-formulas document at
http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/pie_usb-calibration.odt
. By the way, the capability to override calibration values with values
read from text files is still in the backend, just look for pie_usb_poke
in pie.c.
Shading correction was easy because the scanner tells in the calibration
read record what brightness he was aiming at (Jan's Auto-tuned
full-scale-level values) for the calibration lines. So, as one knows the
outcome one can correct. Calibration lines are always read at full
resolution, i.e. every sensor element is used. For a real scan at lower
resolution the block of the COPY (0x18) command tells which sensor
elements were used so that one can correct each pixel.

I do not have to elaborate the dirt cleaning by infrared here because
the pie_usb_sw_post routine in pie.c follows the outline already given
in
http://lists.alioth.debian.org/pipermail/sane-devel/2011-July/028810.html .

Attached patch includes all this. The patch can be applied to current
SANE git 73d450d Sat Aug 18 23:00:14 2012. After a autoreconf [--force]
and configure ...[--enable-pthread] it can be build copied and used.
Although I tried to minimize the diff it remains rather unreadable for
me. Therefore I used doxygen style comments for the pie USB code. The
html output you find at
http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/html .
The parent directory of that contains the patch again, the document on
calibration, the scripts which were used to process the USB snoops and
the doxygen tar ball and a screen shot using XSane.

Now my questions:
Would code like the one in the patch be acceptable in SANE?
Would a separate backend be preferable to patching pie?
What can I do about the license issue already mentioned? I think I could
just ask Emre Celebri to allow SANE's license exception for the three
routines which were adapted from his GPLed code. He seems to be still at
lsus.edu .
Regards
Michael
Vleeshouwers, J.M.
2012-08-23 18:02:11 UTC
Permalink
Hi Michael,

Good to see you picked it up again!

Regarding your 2nd question: I think we should create a separate backend. I've got two reasons for that: 1) the patch makes the pie backend a very large and complex unit, and 2) I don't want to have to worry about breaking the existing pie functionality every time we modify the usb part. I hope you and Klaus agree.

I did some work on this track, although it is not complete. I took your code (version fall 2011) apart and I am now slowly getting all pieces to work again (in a way I understand). The major thing that does not work yet is sane_read(). Your code is based on the index-format, but I think the pixel-format should be much more practical, since the SANE standard expects RGB pixels to be returned from sane_read().
I divided the code into four sections:
a. Low-level functions which communicate with the device on a byte-level. These implement the usb-wrapper but only handle byte-arrays, no interpretation takes place here. This section is complete and tested.
b. Mid-level scanner functions for scanner commands, which take a meaningful struct as an argument (such as mode_select() which takes a 'struct Mode' as a parameter) and return a 'struct Status' which includes sense data if an error occurs. These functions automatically issue a REQUEST SENSE command if a command returns CHECK SENSE, and repeat a command if the device is reports a busy state. No reason to take care of this anymore at a higher level, which is practical. This section is almost complete, and quite easy to extend. Klaus' device has commands for moving to the next and previous slide, these are not implemented (not yet).
c. Top-level auxiliary functions, e.g. for initialising options, correcting shading, etc. A lot still missing.
d. Top-level SANE interface functions. Missing sane_read() and sane_get_parameters(), the rest implemented according to the procedure of the Windows driver. Not much tested yet.

I see you managed to find and implement dust/scratch removal algorithms. That is very welcome! I hope we can get a positive answer to your third question. It looks like you chose to implement this in the backend, which is practical since it does not break the SANE standard. This might be the best solution for now.
This makes our wishlist a bit smaller as far as I am concerned. Remains:
- react to a button press (use scanbuttond daemon? it that the solution we want?)
- export infrared data (for other uses than dust/scratch removal; maybe modify the scanimage frontend and stiff.c to allow RGBI TIFF image files to be exported?)

So how do we proceed?
- We have got two pie-patched backends (Klaus' and yours) which I assume should be merged
- We have got a stand-alone backend in an incomplete state (see attachment)
We should decide on a common starting point. If we decide for a separate backend, I would prefer the four-section structure. You and Klaus are familiar with software development, at least more than I am, so we should use that.

Yours,

Jan

-----Original Message-----
From: Michael Rickmann [mailto:***@gwdg.de]
Sent: dinsdag 21 augustus 2012 22:27
To: sane-***@lists.alioth.debian.org; Vleeshouwers, J.M.
Subject: Reflecta ProScan / Crystalscan 7200 PIE film scanner update

It has become a bit silent about these scanners since Jan Vleeshouwers'
post 8 months ago (
http://lists.alioth.debian.org/pipermail/sane-devel/2011-December/029337.html
). Finally, I had some time to collect what else Jan and I had found out last year and to put it into a patch against the pie backend. Both scanners identify as PIE SF Scanners. They share the same USB id
0x05e3:0x0145 use a SCSI over USB protocol and can be distinguished by byte 0x74 in the INQUIRY return block. Support for the ProScan is quite reasonable now, for the Crystalscan presumably less because calibration (see below) was mostly guessed from what Jan had snooped and communicated. I tested with scanimage and XSane. Currently, Klaus Kaempf seems to work on a PIE MF Scanner ( http://lists.alioth.debian.org/pipermail/sane-devel/2012-July/030092.html ) which is more advanced but presumably not so much different in protocol.
What can the backend do with the SF scanners hardware?
1) scan at resolutions from 20 to 3600 dpi, increment 1
2) read RGB and single pass RGBI data
3) read at 8 and 16 bit [recommended] colour depth
4) calibrate and read in normal and quality [recommended] mode This is what could be detected in snoops using Cyberview and Silverfast under Windows. Silverfast does not support normal mode. Calibration and shading correction is done with the help of software.
What can the backend do by processing the data read?
1) option: clean the infrared colour plane from red spectral overlay
2) option: use the infrared data to remove dirt, output RGB
3) option: simple film grain smoothening (just a triangular blur)
4) option: crop using sanei_magic except for cleaned scans of positive film where an internal routine detects faster and more correctly
5) options: simple colour conversions, constant gamma, invert Scanned positive film really looks good, negative film bad due to still missing colour profiles. Also the options of Xsane for negative film do not improve things. Scanning a 36x24 mm slide at 3600 dpi takes about 88 sec in RGB, 102 sec in RGBI with infrared discarded, 110 sec in RGBI with cleaning and cropping. If one compares these numbers with the ones at http://www.filmscanner.info/en/ReflectaProScan7200.html it seems that the backend handles the scanner correctly.

What can the backend do in spite of the USB additions? Recently I got hold of a 12 year old Adlib JetScan 630 SCSI flatbed scanner which is supported by the pie backend. The scanner still scans at 600 dpi (its
maximum) at “nomal” and “fine” speed; colour, grayscale, halftone and lineart are ok. However at high resolutions and “pro” speed I get hardware “integration time adjustment failure”s “too dark” or “too light”. I get the same errors with SANE-Backends-1.0.22 as included in Ubuntu 12.04 64-bit. I guess, it is the age or a limitation of this kind of scanner. So I think that the direct SCSI part of the backend has not deteriorated.

Description of the added code: I have added three files pie_usb.h contains basic definitions for the USB code sanei_ir.h, sanei_ir.c contain code to remove dirt with help of the infrared plane. I think that this code can migrate to sanei_magic because it may become useful also for other backends. There is still a license problem. I have adapted three threshold routines from M. Emre Celebi's Fourier_0.8 project ( http://sourceforge.net/projects/fourier-ipal/ ) which is licensed under the GNU General Public License Version 2.
Changed files:
sanei_magic.c : sanei_magic_crop can crop images with >8 bit colour depth now pie_scsidefs.h : mainly added support for PIE vendor specific SCSI commands.
pie.c : really a lot. All pure USB functions are prefixed with pie_usb_, all other functions with pie_ (renamed a few) or sane_. The SCSI over USB emulation runs via pie_usb_scsi_wrapper which branches into low level USB code. All other functions talk SCSI. As pie_usb_scsi_wrapper uses the same arguments as sanei_scsi_cmd functions can use a function pointer stored in the scanner structure to do direct SCSI or SCSI over USB. Another possibility to distinguish between the two possibilities is the “model” variable, also in the scanner structure. If it is NULL it indicates a SCSI scanner, otherwise it contains a pointer to the description of an USB scanner. A number of the original functions had become so messed up with if-thens that I wrote pie_usb_ counterparts to ensure that the SCSI part of the backend was not suffering.
A note on calibration:
An outline of what happens during a scan you can find in the introductory comment of pie_usb.h. The best reference for the commands and what they contain is in Jan's post cited above. The scanners use two custom SCSI commands for reading and sending calibration records. During full calibration a record is read from the scanner, then a few calibration lines are read and last, altered calibration data is sent back to the scanner. Although Jan and I could figure out the meaning of most record fields, the snoops did not tell us what was happening between reading and sending in the Windows software. Reverse engineering seemed hopeless. I know Jan would like to have an exact calibration procedure derived from data sheets. There is one huge custom made chip in PIE scanners which presumably does the essential things, and of course there is no documentation. So I started a life-science approach by empirically determining a mathematical model and its parameters. Once one knows what to send for exposure time, gain and offset, one can try different values, scan an empty slide and measure mean color values of the pictures. It melts down to find a relation between exposure time and gain and resulting brightnesses for R, G, B because the Windows programs never changed offset or infrared values between calibration reading and writing. In OpenOffice Calc one can fit a number of curves to the time, gain an illumination values to get a first idea or one can compare the measured values to ones of an own function. In the end the functions giving the best fit were highly non-linear and their parameters had to be determined with help of OpenOffice's evolutionary solver. For a short description of how calibration is done look at the comments before the pie_usb_cali functions in pie.c. If that is not sufficient look at the pie-usb-calibration-formulas document at http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/pie_usb-calibration.odt
. By the way, the capability to override calibration values with values read from text files is still in the backend, just look for pie_usb_poke in pie.c.
Shading correction was easy because the scanner tells in the calibration read record what brightness he was aiming at (Jan's Auto-tuned full-scale-level values) for the calibration lines. So, as one knows the outcome one can correct. Calibration lines are always read at full resolution, i.e. every sensor element is used. For a real scan at lower resolution the block of the COPY (0x18) command tells which sensor elements were used so that one can correct each pixel.

I do not have to elaborate the dirt cleaning by infrared here because the pie_usb_sw_post routine in pie.c follows the outline already given in http://lists.alioth.debian.org/pipermail/sane-devel/2011-July/028810.html .

Attached patch includes all this. The patch can be applied to current SANE git 73d450d Sat Aug 18 23:00:14 2012. After a autoreconf [--force] and configure ...[--enable-pthread] it can be build copied and used.
Although I tried to minimize the diff it remains rather unreadable for me. Therefore I used doxygen style comments for the pie USB code. The html output you find at http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/html .
The parent directory of that contains the patch again, the document on calibration, the scripts which were used to process the USB snoops and the doxygen tar ball and a screen shot using XSane.

Now my questions:
Would code like the one in the patch be acceptable in SANE?
Would a separate backend be preferable to patching pie?
What can I do about the license issue already mentioned? I think I could just ask Emre Celebri to allow SANE's license exception for the three routines which were adapted from his GPLed code. He seems to be still at lsus.edu .
Regards
Michael
m. allan noah
2012-08-23 18:20:09 UTC
Permalink
I've not had time to review the code, but It sounds like you guys are
on the right track. You now know more about the pie backend that most
people, so you are in the best position to judge the common vs
separate backend issue. Either is choice is acceptable. Regarding the
license, the sane exception specifically allows that the exception be
removed by later authors. The only question would be another backend
that chooses to use your IR library. Would that backend also have to
remove the exeption? Probably so, but this is a gray area.

allan

On Thu, Aug 23, 2012 at 2:02 PM, Vleeshouwers, J.M.
<***@tue.nl> wrote:
> Hi Michael,
>
> Good to see you picked it up again!
>
> Regarding your 2nd question: I think we should create a separate backend. I've got two reasons for that: 1) the patch makes the pie backend a very large and complex unit, and 2) I don't want to have to worry about breaking the existing pie functionality every time we modify the usb part. I hope you and Klaus agree.
>
> I did some work on this track, although it is not complete. I took your code (version fall 2011) apart and I am now slowly getting all pieces to work again (in a way I understand). The major thing that does not work yet is sane_read(). Your code is based on the index-format, but I think the pixel-format should be much more practical, since the SANE standard expects RGB pixels to be returned from sane_read().
> I divided the code into four sections:
> a. Low-level functions which communicate with the device on a byte-level. These implement the usb-wrapper but only handle byte-arrays, no interpretation takes place here. This section is complete and tested.
> b. Mid-level scanner functions for scanner commands, which take a meaningful struct as an argument (such as mode_select() which takes a 'struct Mode' as a parameter) and return a 'struct Status' which includes sense data if an error occurs. These functions automatically issue a REQUEST SENSE command if a command returns CHECK SENSE, and repeat a command if the device is reports a busy state. No reason to take care of this anymore at a higher level, which is practical. This section is almost complete, and quite easy to extend. Klaus' device has commands for moving to the next and previous slide, these are not implemented (not yet).
> c. Top-level auxiliary functions, e.g. for initialising options, correcting shading, etc. A lot still missing.
> d. Top-level SANE interface functions. Missing sane_read() and sane_get_parameters(), the rest implemented according to the procedure of the Windows driver. Not much tested yet.
>
> I see you managed to find and implement dust/scratch removal algorithms. That is very welcome! I hope we can get a positive answer to your third question. It looks like you chose to implement this in the backend, which is practical since it does not break the SANE standard. This might be the best solution for now.
> This makes our wishlist a bit smaller as far as I am concerned. Remains:
> - react to a button press (use scanbuttond daemon? it that the solution we want?)
> - export infrared data (for other uses than dust/scratch removal; maybe modify the scanimage frontend and stiff.c to allow RGBI TIFF image files to be exported?)
>
> So how do we proceed?
> - We have got two pie-patched backends (Klaus' and yours) which I assume should be merged
> - We have got a stand-alone backend in an incomplete state (see attachment)
> We should decide on a common starting point. If we decide for a separate backend, I would prefer the four-section structure. You and Klaus are familiar with software development, at least more than I am, so we should use that.
>
> Yours,
>
> Jan
>
> -----Original Message-----
> From: Michael Rickmann [mailto:***@gwdg.de]
> Sent: dinsdag 21 augustus 2012 22:27
> To: sane-***@lists.alioth.debian.org; Vleeshouwers, J.M.
> Subject: Reflecta ProScan / Crystalscan 7200 PIE film scanner update
>
> It has become a bit silent about these scanners since Jan Vleeshouwers'
> post 8 months ago (
> http://lists.alioth.debian.org/pipermail/sane-devel/2011-December/029337.html
> ). Finally, I had some time to collect what else Jan and I had found out last year and to put it into a patch against the pie backend. Both scanners identify as PIE SF Scanners. They share the same USB id
> 0x05e3:0x0145 use a SCSI over USB protocol and can be distinguished by byte 0x74 in the INQUIRY return block. Support for the ProScan is quite reasonable now, for the Crystalscan presumably less because calibration (see below) was mostly guessed from what Jan had snooped and communicated. I tested with scanimage and XSane. Currently, Klaus Kaempf seems to work on a PIE MF Scanner ( http://lists.alioth.debian.org/pipermail/sane-devel/2012-July/030092.html ) which is more advanced but presumably not so much different in protocol.
> What can the backend do with the SF scanners hardware?
> 1) scan at resolutions from 20 to 3600 dpi, increment 1
> 2) read RGB and single pass RGBI data
> 3) read at 8 and 16 bit [recommended] colour depth
> 4) calibrate and read in normal and quality [recommended] mode This is what could be detected in snoops using Cyberview and Silverfast under Windows. Silverfast does not support normal mode. Calibration and shading correction is done with the help of software.
> What can the backend do by processing the data read?
> 1) option: clean the infrared colour plane from red spectral overlay
> 2) option: use the infrared data to remove dirt, output RGB
> 3) option: simple film grain smoothening (just a triangular blur)
> 4) option: crop using sanei_magic except for cleaned scans of positive film where an internal routine detects faster and more correctly
> 5) options: simple colour conversions, constant gamma, invert Scanned positive film really looks good, negative film bad due to still missing colour profiles. Also the options of Xsane for negative film do not improve things. Scanning a 36x24 mm slide at 3600 dpi takes about 88 sec in RGB, 102 sec in RGBI with infrared discarded, 110 sec in RGBI with cleaning and cropping. If one compares these numbers with the ones at http://www.filmscanner.info/en/ReflectaProScan7200.html it seems that the backend handles the scanner correctly.
>
> What can the backend do in spite of the USB additions? Recently I got hold of a 12 year old Adlib JetScan 630 SCSI flatbed scanner which is supported by the pie backend. The scanner still scans at 600 dpi (its
> maximum) at “nomal” and “fine” speed; colour, grayscale, halftone and lineart are ok. However at high resolutions and “pro” speed I get hardware “integration time adjustment failure”s “too dark” or “too light”. I get the same errors with SANE-Backends-1.0.22 as included in Ubuntu 12.04 64-bit. I guess, it is the age or a limitation of this kind of scanner. So I think that the direct SCSI part of the backend has not deteriorated.
>
> Description of the added code: I have added three files pie_usb.h contains basic definitions for the USB code sanei_ir.h, sanei_ir.c contain code to remove dirt with help of the infrared plane. I think that this code can migrate to sanei_magic because it may become useful also for other backends. There is still a license problem. I have adapted three threshold routines from M. Emre Celebi's Fourier_0.8 project ( http://sourceforge.net/projects/fourier-ipal/ ) which is licensed under the GNU General Public License Version 2.
> Changed files:
> sanei_magic.c : sanei_magic_crop can crop images with >8 bit colour depth now pie_scsidefs.h : mainly added support for PIE vendor specific SCSI commands.
> pie.c : really a lot. All pure USB functions are prefixed with pie_usb_, all other functions with pie_ (renamed a few) or sane_. The SCSI over USB emulation runs via pie_usb_scsi_wrapper which branches into low level USB code. All other functions talk SCSI. As pie_usb_scsi_wrapper uses the same arguments as sanei_scsi_cmd functions can use a function pointer stored in the scanner structure to do direct SCSI or SCSI over USB. Another possibility to distinguish between the two possibilities is the “model” variable, also in the scanner structure. If it is NULL it indicates a SCSI scanner, otherwise it contains a pointer to the description of an USB scanner. A number of the original functions had become so messed up with if-thens that I wrote pie_usb_ counterparts to ensure that the SCSI part of the backend was not suffering.
> A note on calibration:
> An outline of what happens during a scan you can find in the introductory comment of pie_usb.h. The best reference for the commands and what they contain is in Jan's post cited above. The scanners use two custom SCSI commands for reading and sending calibration records. During full calibration a record is read from the scanner, then a few calibration lines are read and last, altered calibration data is sent back to the scanner. Although Jan and I could figure out the meaning of most record fields, the snoops did not tell us what was happening between reading and sending in the Windows software. Reverse engineering seemed hopeless. I know Jan would like to have an exact calibration procedure derived from data sheets. There is one huge custom made chip in PIE scanners which presumably does the essential things, and of course there is no documentation. So I started a life-science approach by empirically determining a mathematical model and its parameters. Once one knows what to send for exposure time, gain and offset, one can try different values, scan an empty slide and measure mean color values of the pictures. It melts down to find a relation between exposure time and gain and resulting brightnesses for R, G, B because the Windows programs never changed offset or infrared values between calibration reading and writing. In OpenOffice Calc one can fit a number of curves to the time, gain an illumination values to get a first idea or one can compare the measured values to ones of an own function. In the end the functions giving the best fit were highly non-linear and their parameters had to be determined with help of OpenOffice's evolutionary solver. For a short description of how calibration is done look at the comments before the pie_usb_cali functions in pie.c. If that is not sufficient look at the pie-usb-calibration-formulas document at http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/pie_usb-calibration.odt
> . By the way, the capability to override calibration values with values read from text files is still in the backend, just look for pie_usb_poke in pie.c.
> Shading correction was easy because the scanner tells in the calibration read record what brightness he was aiming at (Jan's Auto-tuned full-scale-level values) for the calibration lines. So, as one knows the outcome one can correct. Calibration lines are always read at full resolution, i.e. every sensor element is used. For a real scan at lower resolution the block of the COPY (0x18) command tells which sensor elements were used so that one can correct each pixel.
>
> I do not have to elaborate the dirt cleaning by infrared here because the pie_usb_sw_post routine in pie.c follows the outline already given in http://lists.alioth.debian.org/pipermail/sane-devel/2011-July/028810.html .
>
> Attached patch includes all this. The patch can be applied to current SANE git 73d450d Sat Aug 18 23:00:14 2012. After a autoreconf [--force] and configure ...[--enable-pthread] it can be build copied and used.
> Although I tried to minimize the diff it remains rather unreadable for me. Therefore I used doxygen style comments for the pie USB code. The html output you find at http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/html .
> The parent directory of that contains the patch again, the document on calibration, the scripts which were used to process the USB snoops and the doxygen tar ball and a screen shot using XSane.
>
> Now my questions:
> Would code like the one in the patch be acceptable in SANE?
> Would a separate backend be preferable to patching pie?
> What can I do about the license issue already mentioned? I think I could just ask Emre Celebri to allow SANE's license exception for the three routines which were adapted from his GPLed code. He seems to be still at lsus.edu .
> Regards
> Michael
>
>
> --
> sane-devel mailing list: sane-***@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/sane-devel
> Unsubscribe: Send mail with subject "unsubscribe your_password"
> to sane-devel-***@lists.alioth.debian.org



--
"The truth is an offense, but not a sin"
Vleeshouwers, J.M.
2012-09-06 21:41:25 UTC
Permalink
Hi Michael & Klaus,

A working separate Reflecta/PIE-USB demo backend in the attachment. At least it works for a Crystalscan 7200. I'm curious about your devices.

The backend consists of:
* reflecta.c: the SANE interface implementation
* reflecta_specific.c: several auxiliary functions
* reflecta_scancmd.c: scanner command functions
* reflecta_usb.c: usb level functions
* reflecta_buffer.c: reader buffer (queue)
and corresponding header files.

I added a modified scanimage frontend, to allow 4-channel TIFF generation. The frontend is hardcoded for reflecta only, and consists of:
* scanimage00.c: modified, see below
* stiff.c: added a 4-channel TIFF header function
* sanei_usb.c: the file from the regular distribution
* sane.h: disabled the definitions of sane_xxx (they move to reflecta.h); enabled SANE_STATUS_WARMING_UP and SANE_FRAME_RGBI
The modifications to scanimage are:
* all functions sane_xxx changed to sane_reflecta_xxx
* gain and offset are set using array syntax; scanimage already has the capability to do that, but it is not documented (so I added a couple of lines to the help text as well)
* added reading RGBI data from the scanner (and save it as 4-channel TIFF)

Build the modified reflecta-frontend+backend:
* include directories: sane-backends-git20120722/backend sane-backends-git20120722/include /usr/include/libusb-1.0
* libraries: libsane & libusb-1.0

I did only shallow testing:
./scanimage -L
./scanimage -A
./scanimage --test
./scanimage --format=pnm >test00.pnm
./scanimage --format=tiff test01.tiff
./scanimage --mode=Color+Infrared --format=tiff >test02.tiff

The list of current options:
Scan Mode:
--mode Lineart|Halftone|Gray|Color|Color+Infrared [Color]
Selects the scan mode (e.g., lineart, monochrome, or color).
--depth 1|8|12|16 [8]
Number of bits per sample, typical values are 1 for "line-art" and 8
for multibit scans.
--resolution 25..7200dpi (in steps of 1) [300]
Sets the resolution of the scanned image.
Geometry:
-l 0..37.6767mm [0]
Top-left x position of scan area.
-t 0..24.2993mm [0]
Top-left y position of scan area.
-x 0..37.6767mm [37.6767]
Width of scan-area.
-y 0..24.2993mm [24.2993]
Height of scan-area.
Enhancement:
--halftone-pattern 53lpi 45d ROUND|70lpi 45d ROUND|75lpi Hori. Line|4X4 BAYER|
4X4 SCROLL|5x5 26 Levels|4x4 SQUARE|5x5 TILE [inactive]
Defines the halftoning (dithering) pattern for scanning halftoned
images.
--threshold 0..100% [inactive]
Select minimum-brightness to get a white point
--sharpen[=(yes|no)] [yes]
Sharpen scan by taking more time to discharge the CCD.
--skip-calibration[=(yes|no)] [no]
Skip auto-calibration before scanning image. Option may be overridden
by scanner.
--fast-infrared[=(yes|no)] [no]
Do not reposition scan head before scanning infrared line. Results in
an infrared offset which may deteriorate IR dust and scratch removal.
Advanced:
--preview[=(yes|no)] [no]
Request a preview-quality scan.
--save-shading-data[=(yes|no)] [no]
Save shading data in 'reflecta.shading'
--save-ccdmask[=(yes|no)] [no]
Save CCD mask 'reflecta.ccd'
--exposure-time 100..5000us (in steps of 1) [2937, 2937, 2937, 2937]
The time the 4 different color filters of the CCD are exposed
(R,G,B,I)
--gain 0..63 [19, 19, 19, 19]
The gain of the signal processor for the 4 CCD color filters (R,G,B,I)
--offset 0..255 [0, 0, 0, 0]
The offset of the signal processor for the 4 CCD color filters
(R,G,B,I)

Yours,

Jan
Klaus Kaempf
2012-09-07 19:54:19 UTC
Permalink
Hi Jan,

thanks a lot for sharing your code. I pushed it to
git://github.com/kkaempf/sane-backends.git, branch "reflecta".


* Vleeshouwers, J.M. <***@tue.nl> [Sep 06. 2012 23:41]:
> Hi Michael & Klaus,
>
> A working separate Reflecta/PIE-USB demo backend in the attachment. At least it works for a Crystalscan 7200. I'm curious about your devices.

I couldn't get it compiled since reflecta7200.h is missing.
I'm also unsure about the required changes to backend/Makefile.am (see
https://github.com/kkaempf/sane-backends/commit/4505161aff70d295c4578c69f0592eeeb0712d18)


Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Vleeshouwers, J.M.
2012-09-07 20:54:08 UTC
Permalink
Hi Klaus,

The reference to reflecta7200.h (in reflecta.c line 42) is a leftover from previous work. It does nothing, just remove the line.
See below for the things I needed to do to add the backend. This is a summary of notes I took when trying to get it to work, your system may be different but I suppose it will at least be similar. Hope it helps.

Yours,

Jan


-----

Description for udev:
copy .desc file to ~/Source/sane-backends-git20120722/tools/udev
running ~/Source/sane-backends-git20120722/tools$ ./sane-desc -s udev -m udev
gives output for rules file; merge these entries with libsane.rules (/lib/udev/rules.d/40-libsane.rules)
(is there an other way if the other .desc-files are not present?
File made for backend, see tools subdirectory.

Dynamic loader:
/etc/sane.d/dll.conf - Configuration file for the SANE dynamic backend loader => add reflecta!
If not, the backend lib will not be found dynamically.

Other modifications:
1. in include/sane/sane.h:74: #if 1 to include status SANE_STATUS_WARMING_UP
2. in include/sane/sane.h:185: #if 1 to include SANE_FRAME_IR and SANE_FRAME_RGBI

Shared library building:
1. backends/Makefile.am: add rules for creating reflecta library (libsane-reflecta.la and the like)
Following the instructions in the make file:
line 80: BACKEND_CONFS: add reflecta.conf
line 157: be_convenience_libs: add libreflecta.la
line 200: be_dlopen_libs: add libsane-reflecta.la
line 240/249: add lines
libreflecta_la_SOURCES = reflecta.c reflecta.h
libreflecta_la_CPPFLAGS = $(AM_CPPFLAGS) -DBACKEND_NAME=reflecta
nodist_libsane_reflecta_la_SOURCES = reflecta-s.c
libsane_reflecta_la_CPPFLAGS = $(AM_CPPFLAGS) -DBACKEND_NAME=reflecta
libsane_reflecta_la_LDFLAGS = $(DIST_SANELIBS_LDFLAGS)
libsane_reflecta_la_LIBADD = $(COMMON_LIBS) libreflecta.la ../sanei/sanei_init_debug.lo ../sanei/sanei_constrain_value.lo ../sanei/sanei_config.lo ../sanei/sanei_config2.lo ../sanei/sanei_usb.lo ../sanei/sanei_scsi.lo ../sanei/sanei_thread.lo sane_strstatus.lo $(USB_LIBS)
EXTRA_DIST += reflecta.conf.in

2. Generate Makefile.in from it by:
rm backends/Makefile.in
cd ~/Source/sane-backends-git20120722
automake --add-missing
(also needed to recreate aclocal.m4: run aclocal without parameters)

3. Then rerun 'configure' and 'make'
BACKENDS=reflecta ./configure --prefix=/usr --sysconfdir=/etc --disable-locking --enable-libusb_1_0

4. Build the backend
cd ~/Source/sane-backends-git20120722
make


________________________________________
From: Klaus Kaempf [***@suse.de]
Sent: Friday, September 07, 2012 9:54 PM
To: Vleeshouwers, J.M.
Cc: 'Michael Rickmann'; sane-***@lists.alioth.debian.org
Subject: Re: Reflecta ProScan / Crystalscan 7200 PIE film scanner update

Hi Jan,

thanks a lot for sharing your code. I pushed it to
git://github.com/kkaempf/sane-backends.git, branch "reflecta".


* Vleeshouwers, J.M. <***@tue.nl> [Sep 06. 2012 23:41]:
> Hi Michael & Klaus,
>
> A working separate Reflecta/PIE-USB demo backend in the attachment. At least it works for a Crystalscan 7200. I'm curious about your devices.

I couldn't get it compiled since reflecta7200.h is missing.
I'm also unsure about the required changes to backend/Makefile.am (see
https://github.com/kkaempf/sane-backends/commit/4505161aff70d295c4578c69f0592eeeb0712d18)


Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Michael Rickmann
2012-09-13 08:55:31 UTC
Permalink
Hi Jan, hello Klaus,
first I tried to clone Jan's backend from Klaus repository at github but
failed to reconfigure it. I really tried hard something in the autofiles
was always missing. In the end I had to copy some files. Then I applied
Jan's files to a recent sane-git with the exception of reflecta.conf and
made the necessary changes for a new backend in SANE's build system.
After an autoreconf I configured with
configure --prefix=/usr --libdir=/usr/lib/x86_64-linux-gnu
--sysconfdir=/etc --localstatedir=/var --with-snmp=no --disable-locking
--enable-static . Finally I got it made, still about 130 lines of
warnings. I have not yet compiled the custom scanimage yet. I also
refrained from changing anything in SANE except for the new backend
(renamed the stiff stuff to reflecta_stiff, put the infrared related
defines into the reflecta files).

After making and installing the reflecta and genesys libs, Jan's
reflecta.conf and ... I ran the usual tests under Ubuntu 12.04 64-bit
having the Reflecta and a Canon LiDE scanner plugged in.

***@velvet:~/build/proscan/git$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 002: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer
Bus 004 Device 002: ID 0d8c:000c C-Media Electronics, Inc. Audio Adapter
Bus 003 Device 002: ID 0471:0311 Philips (or NXP) PCVC740K ToUcam Pro [pwc]
Bus 003 Device 003: ID 045e:00f9 Microsoft Corp. Wireless Desktop
Receiver 3.1
Bus 001 Device 005: ID 05e3:0145 Genesys Logic, Inc.
Bus 002 Device 003: ID 04a9:190a Canon, Inc. CanoScan LiDE 210

***@velvet:~/build/proscan/git$ sane-find-scanner

# sane-find-scanner will now attempt to detect your scanner. If the
# result is different from what you expected, first make sure your
# scanner is powered up and properly connected to your computer.

# No SCSI scanners found. If you expected something different, make
sure that
# you have loaded a kernel SCSI driver for your SCSI adapter.

found USB scanner (vendor=0x0471, product=0x0311) at libusb:003:002
found USB scanner (vendor=0x04a9 [Canon], product=0x190a [CanoScan],
chip=GL124) at libusb:002:003
found USB scanner (vendor=0x05e3, product=0x0145, chip=GL842?) at
libusb:001:005
# Your USB scanner was (probably) detected. It may or may not be
supported by
# SANE. Try scanimage -L and read the backend's manpage.
......

***@velvet:~/build/proscan/git$ scanimage -L
[sanei_debug] Setting debug level of reflecta to 255.
[reflecta] sane_init() build 1
[reflecta] sane_init() trying usb 0x05e3 0x0145 0x30
[reflecta] sane_init() trying usb 0x05e3 0x0145 0x36
[reflecta] sane_init() looking for Reflecta scanner 05e3 0145 model 30
[reflecta] find_device_callback: libusb:001:005
[reflecta] find_device_callback: sanei_usb_open failed
[reflecta] sane_init() looking for Reflecta scanner 05e3 0145 model 36
[reflecta] find_device_callback: libusb:001:005
[reflecta] find_device_callback: sanei_usb_open failed
[reflecta] sane_get_devices
[sanei_debug] Setting debug level of genesys to 255.
[genesys] SANE Genesys backend version 1.0 build 2302 from sane-backends
1.0.24git
[genesys] SANE Genesys backend built with libusb
[genesys] sane_init: authorize != null
[genesys] sane_init: little endian machine
[genesys] probe_genesys_devices start
[genesys] attach: start: devp != NULL, may_wait = 0
[genesys] attach: trying to open device `libusb:002:003'
[genesys] attach: device `libusb:002:003' successfully opened
[genesys] attach: found Canon flatbed scanner LiDE 210 at libusb:002:003
[genesys] attach completed
[genesys] probe_genesys_devices completed
[genesys] sane_genesys_init completed
[genesys] sane_get_devices: start: local_only = false
[genesys] probe_genesys_devices start
[genesys] attach: start: devp != NULL, may_wait = 0
[genesys] attach: device `libusb:002:003' was already in device list
[genesys] probe_genesys_devices completed
[genesys] check_present: libusb:002:003 detected.
[genesys] sane_genesys_get_devices completed
device `genesys:libusb:002:003' is a Canon LiDE 210 flatbed scanner
[reflecta] sane_exit()
[genesys] sane_genesys_exit start
[genesys] sane_genesys_exit completed

The result is the same for root and a normal user. Permissions in
/dev/bus/usb/xxx/yyy are allright. The genesys backend from SANE git
works, the reflecta one not. Then I tried different USB ports and once I
got:

***@velvet:~/build/proscan/git$ scanimage -L
[sanei_debug] Setting debug level of reflecta to 255.
[reflecta] sane_init() build 1
[reflecta] sane_init() trying usb 0x05e3 0x0145 0x30
[reflecta] sane_init() trying usb 0x05e3 0x0145 0x36
[reflecta] sane_init() looking for Reflecta scanner 05e3 0145 model 30
[reflecta] find_device_callback: libusb:002:004
[reflecta] cmdGetScannerProperties()
[reflecta] commandScannerRepeat(): enter, repeat=5
[reflecta] commandScannerRepeat(): ready, tries=1
[reflecta] cmdGetScannerProperties()
[reflecta] commandScannerRepeat(): enter, repeat=5
[reflecta] commandScannerRepeat(): ready, tries=1
[reflecta] find_device_callback: wrong model number 54
[reflecta] sane_init() looking for Reflecta scanner 05e3 0145 model 36
[reflecta] find_device_callback: libusb:002:004
[reflecta] find_device_callback: sanei_usb_open failed
[reflecta] sane_get_devices

I really think that we should not invent a custom format for
reflecta.conf but adhere to SANE's standard which is really safe. We
know which scanners we want to support. So there is no need to build
this list dynamically.
static Reflecta_Model crystalscan_7200_model = {
"PIE/Reflecta", /* Device vendor string */
"CrystalScan 7200", /* Device model name */
0x05e3, /* Vendor ID */
0x0145, /* Product ID */
0x30, /* Model ID */
......
}
.....
static Reflecta_Model *reflecta_supported_model_list[] = {
&crystalscan_7200_model, /* Reflecta CrystalScan 7200, id 0x30 */
&proscan_7200_model, /* Reflecta ProScan 7200, id 0x36 */
...,
NULL
};
Something like that, the compiler will do it for us. Then we should not
read the reflecta.conf ourselves but use SANE's configuration framework.
First we check whether USB vendor- and product-ids matches any devices
in above list. If yes, it it is safe to issue one INQUIRY, loop through
our list and check the Reflecta model-id. I have attached a completely
untested file (a mixture between your, mine and genesys backend code)
which is just a study to show the interplay between sane_init,
sane_get_devices and sane_open using SANE's configuration framework.
What the initial device query is concerned I would like to derive the
number, width and bit depth of the calibration lines and the halftone
names from the inquiry.

Yours
Michael


Am 06.09.2012 23:41, schrieb Vleeshouwers, J.M.:
> Hi Michael & Klaus,
>
> A working separate Reflecta/PIE-USB demo backend in the attachment. At least it works for a Crystalscan 7200. I'm curious about your devices.
>
> The backend consists of:
> * reflecta.c: the SANE interface implementation
> * reflecta_specific.c: several auxiliary functions
> * reflecta_scancmd.c: scanner command functions
> * reflecta_usb.c: usb level functions
> * reflecta_buffer.c: reader buffer (queue)
> and corresponding header files.
>
> I added a modified scanimage frontend, to allow 4-channel TIFF generation. The frontend is hardcoded for reflecta only, and consists of:
> * scanimage00.c: modified, see below
> * stiff.c: added a 4-channel TIFF header function
> * sanei_usb.c: the file from the regular distribution
> * sane.h: disabled the definitions of sane_xxx (they move to reflecta.h); enabled SANE_STATUS_WARMING_UP and SANE_FRAME_RGBI
> The modifications to scanimage are:
> * all functions sane_xxx changed to sane_reflecta_xxx
> * gain and offset are set using array syntax; scanimage already has the capability to do that, but it is not documented (so I added a couple of lines to the help text as well)
> * added reading RGBI data from the scanner (and save it as 4-channel TIFF)
>
> Build the modified reflecta-frontend+backend:
> * include directories: sane-backends-git20120722/backend sane-backends-git20120722/include /usr/include/libusb-1.0
> * libraries: libsane & libusb-1.0
>
> I did only shallow testing:
> ./scanimage -L
> ./scanimage -A
> ./scanimage --test
> ./scanimage --format=pnm >test00.pnm
> ./scanimage --format=tiff test01.tiff
> ./scanimage --mode=Color+Infrared --format=tiff >test02.tiff
>
> The list of current options:
> Scan Mode:
> --mode Lineart|Halftone|Gray|Color|Color+Infrared [Color]
> Selects the scan mode (e.g., lineart, monochrome, or color).
> --depth 1|8|12|16 [8]
> Number of bits per sample, typical values are 1 for "line-art" and 8
> for multibit scans.
> --resolution 25..7200dpi (in steps of 1) [300]
> Sets the resolution of the scanned image.
> Geometry:
> -l 0..37.6767mm [0]
> Top-left x position of scan area.
> -t 0..24.2993mm [0]
> Top-left y position of scan area.
> -x 0..37.6767mm [37.6767]
> Width of scan-area.
> -y 0..24.2993mm [24.2993]
> Height of scan-area.
> Enhancement:
> --halftone-pattern 53lpi 45d ROUND|70lpi 45d ROUND|75lpi Hori. Line|4X4 BAYER|
> 4X4 SCROLL|5x5 26 Levels|4x4 SQUARE|5x5 TILE [inactive]
> Defines the halftoning (dithering) pattern for scanning halftoned
> images.
> --threshold 0..100% [inactive]
> Select minimum-brightness to get a white point
> --sharpen[=(yes|no)] [yes]
> Sharpen scan by taking more time to discharge the CCD.
> --skip-calibration[=(yes|no)] [no]
> Skip auto-calibration before scanning image. Option may be overridden
> by scanner.
> --fast-infrared[=(yes|no)] [no]
> Do not reposition scan head before scanning infrared line. Results in
> an infrared offset which may deteriorate IR dust and scratch removal.
> Advanced:
> --preview[=(yes|no)] [no]
> Request a preview-quality scan.
> --save-shading-data[=(yes|no)] [no]
> Save shading data in 'reflecta.shading'
> --save-ccdmask[=(yes|no)] [no]
> Save CCD mask 'reflecta.ccd'
> --exposure-time 100..5000us (in steps of 1) [2937, 2937, 2937, 2937]
> The time the 4 different color filters of the CCD are exposed
> (R,G,B,I)
> --gain 0..63 [19, 19, 19, 19]
> The gain of the signal processor for the 4 CCD color filters (R,G,B,I)
> --offset 0..255 [0, 0, 0, 0]
> The offset of the signal processor for the 4 CCD color filters
> (R,G,B,I)
>
> Yours,
>
> Jan=
Klaus Kaempf
2012-09-13 12:49:53 UTC
Permalink
* Michael Rickmann <***@gwdg.de> [Sep 13. 2012 10:55]:
> Hi Jan, hello Klaus,
> first I tried to clone Jan's backend from Klaus repository at github
> but failed to reconfigure it.

Sorry it didn't work for you. I usually remove generated files from git.

After git checkout, you should run (documented here for sane-devel)

- libtoolize
- autoreconf -fi
- BACKENDS=pie ./configure --prefix=/usr --libdir=/usr/lib64 \
--sysconfdir=/etc --localstatedir=/var --enable-libusb_1_0 \
--disable-static
(--libdir is for a 64bit system, change it to /usr/lib for 32bit)

You probably missed the libtoolize part.

[...]
> I have attached a completely untested file (a mixture between your,
> mine and genesys backend code) which is just a study to show the
> interplay between sane_init, sane_get_devices and sane_open using
> SANE's configuration framework.

Great. That was a complete mystery to me when reading the old pie.c
file.

I've checked in your code as branch "mrickma" into my sane-backup
github repo: https://github.com/kkaempf/sane-backends/tree/mrickma

See
https://github.com/kkaempf/sane-backends/commit/20ef5bef944fc1120342c014b2e232a747af48b1
for the changes needed to recognize a Reflecta DigitDia 6000 (should
be identical to PIE PowerSlide 3600) scanner.

Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Michael Rickmann
2012-09-13 13:59:11 UTC
Permalink
Hi Klaus,


Am 13.09.2012 14:49, schrieb Klaus Kaempf:
> * Michael Rickmann <***@gwdg.de> [Sep 13. 2012 10:55]:
>> Hi Jan, hello Klaus,
>> first I tried to clone Jan's backend from Klaus repository at github
>> but failed to reconfigure it.
> Sorry it didn't work for you. I usually remove generated files from git.
That explains it. Sorry, I am a git beginner. I have always used git
update-index --assume-unchanged <file> for the auto files.
>
> After git checkout, you should run (documented here for sane-devel)
>
> - libtoolize
> - autoreconf -fi
> - BACKENDS=pie ./configure --prefix=/usr --libdir=/usr/lib64 \
> --sysconfdir=/etc --localstatedir=/var --enable-libusb_1_0 \
> --disable-static
> (--libdir is for a 64bit system, change it to /usr/lib for 32bit)
>
> You probably missed the libtoolize part.
Thanks, I will try
>
> [...]
>> I have attached a completely untested file (a mixture between your,
>> mine and genesys backend code) which is just a study to show the
>> interplay between sane_init, sane_get_devices and sane_open using
>> SANE's configuration framework.
> Great. That was a complete mystery to me when reading the old pie.c
> file.
I am badly organized at the moment trying to catch up with what I missed
during my vaccation. I forgot the attachment, which I have corrected by
answering my own post.
>
> I've checked in your code as branch "mrickma" into my sane-backup
> github repo: https://github.com/kkaempf/sane-backends/tree/mrickma
>
> See
> https://github.com/kkaempf/sane-backends/commit/20ef5bef944fc1120342c014b2e232a747af48b1
> for the changes needed to recognize a Reflecta DigitDia 6000 (should
> be identical to PIE PowerSlide 3600) scanner.
>
> Regards,
>
> Klaus
Does it really have USB ids 0x05e3, 0x0142 as written in
http://www.sane-project.org/unsupported/reflecta-digitdia-3600.html ?
Jan's and my scanner have 0x05e3, 0x0145. Is there more information
publicly available?
Regards,
Michael
Klaus Kaempf
2012-09-13 14:03:40 UTC
Permalink
* Michael Rickmann <***@gwdg.de> [Sep 13. 2012 15:59]:

> Does it really have USB ids 0x05e3, 0x0142 as written in
> http://www.sane-project.org/unsupported/reflecta-digitdia-3600.html
> ?

Would I lie to you ? :-) Yes 0x5e3:0x0142 is correct for the "Multiple
Slide Scanner" aka "PIE MS Scanner".

> Jan's and my scanner have 0x05e3, 0x0145. Is there more
> information publicly available?

I'm not aware of any. I can send your the debug hex dump of the
INQUIRY result, if that's of any help.

Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Vleeshouwers, J.M.
2012-09-13 19:32:17 UTC
Permalink
Klaus,

Please do.
I'm curious about the model number (byte 116, start counting at 0), but also if there are other differences.
Can you get the whole INQUIRY block (not just the 128 or so the Windows driver uses)?

Jan
________________________________________
From: Klaus Kaempf [***@suse.de]
Sent: Thursday, September 13, 2012 4:03 PM
To: Michael Rickmann
Cc: Vleeshouwers, J.M.; sane-***@lists.alioth.debian.org
Subject: Re: Reflecta ProScan / Crystalscan 7200 PIE film scanner update

* Michael Rickmann <***@gwdg.de> [Sep 13. 2012 15:59]:

> Does it really have USB ids 0x05e3, 0x0142 as written in
> http://www.sane-project.org/unsupported/reflecta-digitdia-3600.html
> ?

Would I lie to you ? :-) Yes 0x5e3:0x0142 is correct for the "Multiple
Slide Scanner" aka "PIE MS Scanner".

> Jan's and my scanner have 0x05e3, 0x0145. Is there more
> information publicly available?

I'm not aware of any. I can send your the debug hex dump of the
INQUIRY result, if that's of any help.

Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Klaus Kaempf
2012-09-16 18:43:09 UTC
Permalink
* Vleeshouwers, J.M. <***@tue.nl> [Sep 13. 2012 21:32]:
> Klaus,
>
> Please do.
> I'm curious about the model number (byte 116, start counting at 0), but also if there are other differences.
> Can you get the whole INQUIRY block (not just the 128 or so the Windows driver uses)?

Attached (hopefully complete).

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG NÌrnberg)
Maxfeldstraße 5, 90409 NÃŒrnberg, Germany
Vleeshouwers, J.M.
2012-09-16 23:05:36 UTC
Permalink
Klaus, Michael,

It's complete, see below.
The differences with the two other scanners are limited:
* I see you have got a square scan area - which make sense
* It has an additinal bit set in options (bit 2)
* Differences in maximum shadow, x0, y0, x1, y1
* Same firmware author but your device is flash
* Model number: 58

Yours,

Jan


0000 06 deviceType: scanner
0001 00 specification0: not removable, no modifications
0002 02 specification0: SCSI-2 compatible
0003 01 specification2: compatibility with some products that were designed prior to the development of this standard
0004 B4 additional length: 180
0005 00 00 00 reserved
0008 50 49 45 20 20 20 20 20 vendorId: 'PIE '
0016 4D 53 20 53 63 61 6E 6E 65 72 00 00 00 00 00 00 productId: 'MS Scanner'
0032 31 2E 30 30 product revision: '1.00'
0036 88 13 max resolution x: 5000
0038 88 13 max resolution y: 5000
0040 18 1D max scanwidth: 7448 (1.49 inch)
0042 D5 1C max scan height: 7381 (1.48 inch)
0044 9E filters: 10011110 = OnePassColor, I, B, G, R
0045 35 color depths: 00110101 = 16, 12, 8, 1
0046 07 color format: 00000111 = Index, Line, Pixel
0047 00 reserved
0048 09 image format: 00001001 = OKLine, Intel
0049 4B scan capability: 01001011 = ExtCal, DisableCal, 3 cal speeds
0050 65 optional: 01100101 = ?, ?, TransModel1, AutoDocDeeder
0051 02 enhancements: 00000010 = ?
0052 0C gamma bits: ?
0053 00 last filter: ?
0054 2C 01 preview resolution: 300
0055 00 00 00 00 00 00 00 00 reserved
0064 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0096 31 2E 30 32 firmware version: 1.02
0100 00 halftones: 0
0101 64 minimum highlight: 100
0102 19 maximum shadow: 25
0103 01 calibration equation: 01 meaning ?
0104 C4 09 maximum exposure: 2500
0106 64 00 minimum exposure: 100
0108 E7 03 x0: 999
0110 47 06 y0: 1607
0112 3B 17 x1: 5947
0114 31 1E y1: 7729
0116 3A model: 58
0117 00 00 00 unknown
0120 50 49 45 00 32 30 31 31 production: 'PIE.2011/06/08 11:30AM'
0128 2F 30 36 2F 30 38 20 31 31 3A 33 30 41 4D 00 00
0144 46 6C 61 73 68 20 52 4F 4D 20 32 39 45 45 35 31 signature: 'Flash ROM 29EE512 \nBY:Chen Tsung-Yu '
0160 32 20 0D 0A 42 59 3A 43 68 65 6E 20 54 73 75 6E
0176 67 2D 59 75 20 20 00 00 64
________________________________________
From: Klaus Kaempf [***@suse.de]
Sent: Sunday, September 16, 2012 8:43 PM
To: Vleeshouwers, J.M.
Cc: Michael Rickmann; sane-***@lists.alioth.debian.org
Subject: Re: Reflecta ProScan / Crystalscan 7200 PIE film scanner update

* Vleeshouwers, J.M. <***@tue.nl> [Sep 13. 2012 21:32]:
> Klaus,
>
> Please do.
> I'm curious about the model number (byte 116, start counting at 0), but also if there are other differences.
> Can you get the whole INQUIRY block (not just the 128 or so the Windows driver uses)?

Attached (hopefully complete).

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Michael Rickmann
2012-09-14 06:52:26 UTC
Permalink
Hi Klaus,

Am 13.09.2012 14:49, schrieb Klaus Kaempf:
>
> After git checkout, you should run (documented here for sane-devel)
>
> - libtoolize
> - autoreconf -fi
> - BACKENDS=pie ./configure --prefix=/usr --libdir=/usr/lib64 \
> --sysconfdir=/etc --localstatedir=/var --enable-libusb_1_0 \
> --disable-static
> (--libdir is for a 64bit system, change it to /usr/lib for 32bit)
>
> You probably missed the libtoolize part.
That helped, thank you. I have attached a patch against your git which
helps to build Jan's backend.
Regards,
Michael
Klaus Kaempf
2012-09-16 18:46:05 UTC
Permalink
* Michael Rickmann <***@gwdg.de> [Sep 14. 2012 08:52]:
> Hi Klaus,
>
> Am 13.09.2012 14:49, schrieb Klaus Kaempf:
> >
> >After git checkout, you should run (documented here for sane-devel)
> >
> > - libtoolize
> > - autoreconf -fi
> > - BACKENDS=pie ./configure --prefix=/usr --libdir=/usr/lib64 \
> > --sysconfdir=/etc --localstatedir=/var --enable-libusb_1_0 \
> > --disable-static
> > (--libdir is for a 64bit system, change it to /usr/lib for 32bit)
> >
> >You probably missed the libtoolize part.
> That helped, thank you. I have attached a patch against your git
> which helps to build Jan's backend.

Thanks. After some more modifications
(https://github.com/kkaempf/sane-backends/commit/038e4244dd7b90d6272786bd2a99bd224545b877)
and adding the device parameters
(https://github.com/kkaempf/sane-backends/commit/f70649ce5a6ead92c99300e9d9b632eeed44e32a)
the 'reflecta' backend recognizes the scanner.

However, it stalls pretty quickly when being run and I had to
power-cycle the device :-/



Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Michael Rickmann
2012-09-13 13:03:29 UTC
Permalink
Hello again,

Am 13.09.2012 10:55, schrieb Michael Rickmann:
> Hi Jan, hello Klaus,
<--- snip
sorry I forgot the attachment.
Michael
Michael Rickmann
2012-09-12 12:01:22 UTC
Permalink
Hi Jan,
a late answer to your questions, just to point out what is not yet
obsolete.

Am 23.08.2012 20:02, schrieb Vleeshouwers, J.M.:
> Hi Michael,
>
> Good to see you picked it up again!
>
> Regarding your 2nd question: I think we should create a separate
> backend. I've got two reasons for that: 1) the patch makes the pie
> backend a very large and complex unit, and 2) I don't want to have to
> worry about breaking the existing pie functionality every time we
> modify the usb part. I hope you and Klaus agree.
>
> I did some work on this track, although it is not complete. I took
> your code (version fall 2011) apart and I am now slowly getting all
> pieces to work again (in a way I understand). The major thing that
> does not work yet is sane_read(). Your code is based on the
> index-format, but I think the pixel-format should be much more
> practical, since the SANE standard expects RGB pixels to be returned
> from sane_read().
I see your point here: The indexed reading results in rather ugly code
because the scanner does not deliver the R, G, B, I colored lines in any
predictable order and one has to store components of several lines
before one is complete. It seems that the amount of data which has to be
buffered can be derived from the filterOffsets in the
Reflecta_Scan_Parameters structure. If you do this in a separate thread
complete lines can be written to a pipe from where they can be
“sane_read”. If you wish to process whole images in software before they
are read there is no need for a separate thread.
Nevertheless, I would like to stick to indexed reading. Arguments follow:
1) The Windows programs use it. None of the snoops I made contained
image data in a different format than INQ_COLOR_FORMAT_INDEX.
2) I discovered reading in INQ_COLOR_FORMAT_LINE by accident because it
was offered by the pie backend and was able to read RGB data. I was not
able to read RGBI. I did not try very hard because there was the other
Windows backed possibility. So work will have to be done here without
being backed by snooping.
3) I suspect that the reordering of the R, G, B, I lines has to be done
somewhere, either in the scanner or in the backend. Certainly the
backend solution would be faster.
> I divided the code into four sections:
> a. Low-level functions which communicate with the device on a
> byte-level. These implement the usb-wrapper but only handle
> byte-arrays, no interpretation takes place here. This section is
> complete and tested.
> b. Mid-level scanner functions for scanner commands, which take a
> meaningful struct as an argument (such as mode_select() which takes a
> 'struct Mode' as a parameter) and return a 'struct Status' which
> includes sense data if an error occurs. These functions automatically
> issue a REQUEST SENSE command if a command returns CHECK SENSE, and
> repeat a command if the device is reports a busy state. No reason to
> take care of this anymore at a higher level, which is practical. This
> section is almost complete, and quite easy to extend. Klaus' device
> has commands for moving to the next and previous slide, these are not
> implemented (not yet).
I like this section very much, especially the internal use of
SANE_STATUS_CHECK_CONDITION. This should be our code.
> c. Top-level auxiliary functions, e.g. for initialising options,
> correcting shading, etc. A lot still missing.
> d. Top-level SANE interface functions. Missing sane_read() and
> sane_get_parameters(), the rest implemented according to the procedure
> of the Windows driver. Not much tested yet.
>
> I see you managed to find and implement dust/scratch removal
> algorithms. That is very welcome! I hope we can get a positive answer
> to your third question. It looks like you chose to implement this in
> the backend, which is practical since it does not break the SANE
> standard. This might be the best solution for now.
> This makes our wishlist a bit smaller as far as I am concerned. Remains:
> - react to a button press (use scanbuttond daemon? it that the
> solution we want?)
> - export infrared data (for other uses than dust/scratch removal;
> maybe modify the scanimage frontend and stiff.c to allow RGBI TIFF
> image files to be exported?)
>
> So how do we proceed?
> - We have got two pie-patched backends (Klaus' and yours) which I
> assume should be merged
> - We have got a stand-alone backend in an incomplete state (see
> attachment)
> We should decide on a common starting point. If we decide for a
> separate backend, I would prefer the four-section structure. You and
> Klaus are familiar with software development, at least more than I am,
> so we should use that.
I guess I am the least professional of us three in this project spending
most time on neurobiology. Ok, four sections plus a fifth one for
processing the read images. Section c should contain task related
routines to gain some fexibility and to keep the SANE interface routines
readable.
>
> Yours,
>
> Jan
>
> -----Original Message-----
> From: Michael Rickmann [mailto:***@gwdg.de]
> Sent: dinsdag 21 augustus 2012 22:27
> To: sane-***@lists.alioth.debian.org; Vleeshouwers, J.M.
> Subject: Reflecta ProScan / Crystalscan 7200 PIE film scanner update
<--- snipp

Regards
Michael
Klaus Kaempf
2012-09-10 13:09:24 UTC
Permalink
Michael,

thanks a lot for posting your changes !

* Michael Rickmann <***@gwdg.de> [Aug 21. 2012 22:28]:
>
> Now my questions:
> Would code like the one in the patch be acceptable in SANE?
> Would a separate backend be preferable to patching pie?

Looking at the amount of changes required to properly support USB PIE
devices and the high risk to break SCSI PIE devices, I'm meanwhile all
for a separate backend.

Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Vleeshouwers, J.M.
2012-09-10 19:01:56 UTC
Permalink
Hi Michael,

I tested your modified Pie backend this weekend. I had to change one line to get it working. My scanner takes about 10 seconds to get ready after it decides to collect shading data after all. So in pie_usb_calibrate() I changed line 3307 to:
status = pie_usb_wait_scanner (scanner, 30);
(30 to be very safe) I also changed the polling frequency of pie_usb_wait_scanner() from 16 times per second to just 1 time per second. The windows driver only polls every 1.5 second. I did not check if this was essential, though.

With this change scanimage -L, -A and -T all work. And although I did not expect it, /tmp contains the scanned and cleaned image and a couple of auxiliary files. I'm impressed, the scratch/dust removal works very well!.

While you are thinking about the proposal to create a separate backend for the Reflecta scanners, I got two more detailed questions for you:
1. When doing shading correction, your routine only corrects 2-byte values >4096 and 1-byte values >16. It looks like an efficiency issue, right? How much faster is it?
2. I did not yet study the dust/scratch removal functions. What seems critical me is how much memory they use, knowing that the Windows driver gets into memory trouble at high resolutions. Do these functions allow processing the data stream from scanner to frontend with limited buffering in between?

Yours,

Jan

________________________________________
From: Klaus Kaempf [***@suse.de]
Sent: Monday, September 10, 2012 3:09 PM
To: Michael Rickmann
Cc: sane-***@lists.alioth.debian.org; Vleeshouwers, J.M.
Subject: Re: [sane-devel] Reflecta ProScan / Crystalscan 7200 PIE film scanner update

Michael,

thanks a lot for posting your changes !

* Michael Rickmann <***@gwdg.de> [Aug 21. 2012 22:28]:
>
> Now my questions:
> Would code like the one in the patch be acceptable in SANE?
> Would a separate backend be preferable to patching pie?

Looking at the amount of changes required to properly support USB PIE
devices and the high risk to break SCSI PIE devices, I'm meanwhile all
for a separate backend.

Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Michael Rickmann
2012-09-12 05:45:31 UTC
Permalink
Hi Jan, hi Klaus,

last weekend came back from vacation without internet and there will be
a similar period starting next week.

Am 10.09.2012 21:01, schrieb Vleeshouwers, J.M.:
> Hi Michael,
>
> I tested your modified Pie backend this weekend. I had to change one line to get it working. My scanner takes about 10 seconds to get ready after it decides to collect shading data after all. So in pie_usb_calibrate() I changed line 3307 to:
> status = pie_usb_wait_scanner (scanner, 30);
> (30 to be very safe) I also changed the polling frequency of pie_usb_wait_scanner() from 16 times per second to just 1 time per second. The windows driver only polls every 1.5 second. I did not check if this was essential, though.

I am glad it worked for you.

> With this change scanimage -L, -A and -T all work. And although I did not expect it, /tmp contains the scanned and cleaned image and a couple of auxiliary files. I'm impressed, the scratch/dust removal works very well!.

Most of my tests were with xsane. These files are written if the debug
level is >= DBG_image. I needed it to control the dust removal. Well, I
am more experienced in manipulating images than in obtaining them. When
we have reached a common basis I will implement something to produce
nice looking images from generic color negative film. The xsane options
for that are rather primitive.

> While you are thinking about the proposal to create a separate backend for the Reflecta scanners, I got two more detailed questions for you:
> 1. When doing shading correction, your routine only corrects 2-byte values >4096 and 1-byte values >16. It looks like an efficiency issue, right? How much faster is it?
I can hardly remember. I think I had a look at the outcome of the rather
simple algorithm and found that at low light intensities the original
values were looking better than the corrected ones, possibly a rouding
issue.
> 2. I did not yet study the dust/scratch removal functions. What seems critical me is how much memory they use, knowing that the Windows driver gets into memory trouble at high resolutions. Do these functions allow processing the data stream from scanner to frontend with limited buffering in between?
These routines rely on a whole image. Currently, they are optimized for
speed. So I hold 4 color planes (~130 MB at 16 bit and 3600 dpi) in
memory, in addition what I need to work with the infrared channel and
the final adaptation of the cleaned pixels. I have not messured but I
guess that the maximum may well be upto 300 MB. The routines work on one
color plane at a time and the RGB assembly of the cleaned images also
takes ~260 MB. I guess, in the end we might strip it down to half
sacrificing speed, by storing the R, G, B color planes in files until
they are needed, using short ints instead of ints and using a separate
thread which immediately pipes the assembled RGB to sane_read.
> Yours,
>
> Jan
>
> ________________________________________
> From: Klaus Kaempf [***@suse.de]
> Sent: Monday, September 10, 2012 3:09 PM
> To: Michael Rickmann
> Cc:sane-***@lists.alioth.debian.org; Vleeshouwers, J.M.
> Subject: Re: [sane-devel] Reflecta ProScan / Crystalscan 7200 PIE film scanner update
>
> Michael,
>
> thanks a lot for posting your changes !
>
> * Michael Rickmann<***@gwdg.de> [Aug 21. 2012 22:28]:
>> Now my questions:
>> Would code like the one in the patch be acceptable in SANE?
>> Would a separate backend be preferable to patching pie?
> Looking at the amount of changes required to properly support USB PIE
> devices and the high risk to break SCSI PIE devices, I'm meanwhile all
> for a separate backend.
Agreed, especially when you think about maintaining it in the long run.
Let us call it reflecta at the moment as the naming in Jan's code seems
to suggest it. Regard my pie patch as a study what can and should be done.
> Regards,
>
> Klaus
> --
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
> Maxfeldstraße 5, 90409 Nürnberg, Germany
I will try to answer some of the questions which have arisen while I was
unresponsive in a different thread.
Yours
Michael
Vleeshouwers, J.M.
2012-09-12 14:03:22 UTC
Permalink
Hi Michael (& Klaus),

Summarizing (from 2 messages):
- Streaming image data to the frontend is not possible, since we want the backend to be able to do dust/scratch removal. Of course there must be an option to retrieve raw RGBI data. So: no streaming. Sane_start() will read all the image data from the scanner into a buffer and do the post-processing to arrive at a complete image. Sane_read() will just return requested parts of the data.
- A separate reader process is not necessary, since the whole image is available as soon as sane_start() finishes .
- We'll stick to the indexed format when reading data from the scanner. No problem, since we already have to buffer internally.
- Memory usage is an issue. Our backend should be able to handle an image of about 0.5 GB (7200 dpi, 16 bit depth, 4 colors), with perhaps the same amount of data for processing. Some resource checks before starting the scan might be welcome, but I don't know how to do that yet.

I'm curious about the dust/scratch-removal. I understand the idea of finding appropriate thresholds to distinguish between image and artifacts, but I get lost in the image correction process ('dilating'). Do you have any literature on this process?

I'm also curious if you have contacted Emre Celebri already about using his algorithms.

Finally: if you are willing to go through my code to find out where your scanner should be treated differently, I shall implement the above, and also try to add the post-processing functions. I was a little too enthusiastic in sending working code (September 6th), because it is not obvious how to get it to compile (sorry, Klaus). I'll be clearer next time, which I hope is before you return from your upcoming internetless time.

Yours,

Jan
Michael Rickmann
2012-09-13 11:33:42 UTC
Permalink
Hi Jan, hello Klaus,

Am 12.09.2012 16:03, schrieb Vleeshouwers, J.M.:
> Hi Michael (& Klaus),
>
> Summarizing (from 2 messages):
> - Streaming image data to the frontend is not possible, since we want the backend to be able to do dust/scratch removal. Of course there must be an option to retrieve raw RGBI data. So: no streaming. Sane_start() will read all the image data from the scanner into a buffer and do the post-processing to arrive at a complete image. Sane_read() will just return requested parts of the data.
Actually there are three cases, a) delivering raw data as for a preview,
b) postprocessing of a whole image where you know the dimensions in
advance, c) auto crop processing. c) has to be done completely in
sane_start, reading for b) can be initiated from sane_read. For a)
sane_start just starts the reader thread and returns. You suggest the b)
case, and I think it is a good starting point. Addition of a) and c) are
not so difficult.
> - A separate reader process is not necessary, since the whole image is available as soon as sane_start() finishes .
See above. It would be nice to see the preview appear in xsane though.
> - We'll stick to the indexed format when reading data from the scanner. No problem, since we already have to buffer internally.
> - Memory usage is an issue. Our backend should be able to handle an image of about 0.5 GB (7200 dpi, 16 bit depth, 4 colors), with perhaps the same amount of data for processing. Some resource checks before starting the scan might be welcome, but I don't know how to do that yet.
Agreed.
> I'm curious about the dust/scratch-removal. I understand the idea of finding appropriate thresholds to distinguish between image and artifacts, but I get lost in the image correction process ('dilating'). Do you have any literature on this process?
Definition: http://homepages.inf.ed.ac.uk/rbf/HIPR2/dilate.htm
Implementation: http://ostermiller.org/dilate_and_erode.html
When you read down the second URL to the Manhattan distances, you will
see that I not only calculated a distance map but also built an index
map which for each dirty pixel contains the linear index of the closest
clean pixel. So replacing dirty pixels becomes trivial and fast.
>
> I'm also curious if you have contacted Emre Celebri already about using his algorithms.
Not yet, but I will.
>
> Finally: if you are willing to go through my code to find out where your scanner should be treated differently, I shall implement the above, and also try to add the post-processing functions. I was a little too enthusiastic in sending working code (September 6th), because it is not obvious how to get it to compile (sorry, Klaus). I'll be clearer next time, which I hope is before you return from your upcoming internetless time.
>
> Yours,
>
> Jan
I think that once we have a minimal backend which works for more than
one of us we could start helping each other. I could fit in calibration,
shading correction, dust removal, cropping as I did it already once.
Regards
Michael
Klaus Kaempf
2012-09-13 14:08:08 UTC
Permalink
* Michael Rickmann <***@gwdg.de> [Sep 13. 2012 13:33]:

> I think that once we have a minimal backend which works for more
> than one of us we could start helping each other.

I'm all for it !

(And github.com is a very nice place for collaborating on git repos.
Hint, hint ;-))

> I could fit in
> calibration, shading correction, dust removal, cropping as I did it
> already once.

Getting these as separate git commits would be awesome for the whole
SANE community.




Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Klaus Kaempf
2012-09-13 12:38:11 UTC
Permalink
Hi Michael,

* Michael Rickmann <***@gwdg.de> [Sep 12. 2012 07:45]:
> Hi Jan, hi Klaus,
>
> last weekend came back from vacation without internet and there will
> be a similar period starting next week.

its great to have you back, but I'm jealous now. ;-)

>
> Am 10.09.2012 21:01, schrieb Vleeshouwers, J.M.:
> >Hi Michael,
> >
> >I tested your modified Pie backend this weekend. I had to change one line to get it working. My scanner takes about 10 seconds to get ready after it decides to collect shading data after all. So in pie_usb_calibrate() I changed line 3307 to:
> >status = pie_usb_wait_scanner (scanner, 30);
> >(30 to be very safe) I also changed the polling frequency of pie_usb_wait_scanner() from 16 times per second to just 1 time per second. The windows driver only polls every 1.5 second. I did not check if this was essential, though.
>
> I am glad it worked for you.

Looks like the Powerslide/DigitDia slide scanner is a bit different.
I couldn't get it to work, scanimage complains about "device busy" -
even after extending the wait period in pie_usb_wait_scanner().

Cranking up the debug level revealed a 'read' request issued without
the scanner indicating availabilty of data. The the scanner hung and I
had to power-cycle it.


Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Michael Rickmann
2012-09-13 16:21:35 UTC
Permalink
Hi Klaus,

Am 13.09.2012 14:38, schrieb Klaus Kaempf:
> Hi Michael,
>
> * Michael Rickmann <***@gwdg.de> [Sep 12. 2012 07:45]:
>> Hi Jan, hi Klaus,
>>
>> last weekend came back from vacation without internet and there will
>> be a similar period starting next week.
> its great to have you back, but I'm jealous now. ;-)
>
>> Am 10.09.2012 21:01, schrieb Vleeshouwers, J.M.:
>>> Hi Michael,
>>>
>>> I tested your modified Pie backend this weekend. I had to change one line to get it working. My scanner takes about 10 seconds to get ready after it decides to collect shading data after all. So in pie_usb_calibrate() I changed line 3307 to:
>>> status = pie_usb_wait_scanner (scanner, 30);
>>> (30 to be very safe) I also changed the polling frequency of pie_usb_wait_scanner() from 16 times per second to just 1 time per second. The windows driver only polls every 1.5 second. I did not check if this was essential, though.
>> I am glad it worked for you.
> Looks like the Powerslide/DigitDia slide scanner is a bit different.
> I couldn't get it to work, scanimage complains about "device busy" -
> even after extending the wait period in pie_usb_wait_scanner().
>
> Cranking up the debug level revealed a 'read' request issued without
> the scanner indicating availabilty of data. The the scanner hung and I
> had to power-cycle it.
>
>
> Klaus
Does that happen during scanimage -L? Without seeing the debug output I
can only guess. Worst case is that your scanner has stalled at the
initial INQUIRY command. I remember from old USB sniffing times that
Cyberview was sending a rather enigmatic initial sequence of control
bytes which could have ment negotiating an address on the SCSI bus. I
tried to mimic them. In Jan's new backend I have seen that that may mean
overdoing things. To get rid of them outcomment the if (cmnd[0] ==
INQUIRY) {} block in pie_usb_scsi_wrapper beginning at line 1236 or try
to get Jan's code working.
The next critical thing is mode setting (pie_usb_mode_select or Jan's
cmdSetMode), especially data[9]. Jan's backend has a clearer definition
of it than my patch. As the scanner may hang later when the calibration
handshake does not fit.
Could you do an export SANE_DEBUG_PIE=255 , scanimage -L 2> dbg0 . If it
succeeds xsane 2> dbg1 , select speed Pro and do a preview. Then post
the debug output which failed.
I am still writing this mail and you have already sent three new ones.
We will get those scanners going.
Regards,
Michael
Klaus Kaempf
2012-09-13 20:22:07 UTC
Permalink
Hi Michael,

* Michael Rickmann <***@gwdg.de> [Sep 13. 2012 18:21]:
> Am 13.09.2012 14:38, schrieb Klaus Kaempf:

> >Cranking up the debug level revealed a 'read' request issued without
> >the scanner indicating availabilty of data. The the scanner hung and I
> >had to power-cycle it.
> >
> >
> >Klaus
> Does that happen during scanimage -L?

Yes.

> Without seeing the debug output I can only guess. Worst case is that your scanner has stalled
> at the initial INQUIRY command.

No, thats not the case. Initial negotiating and calibrating is fine
untile the actual scan starts.

>
Klaus Kaempf
2012-09-13 13:59:49 UTC
Permalink
* Michael Rickmann <***@gwdg.de> [Sep 12. 2012 07:45]:
> Let us call it reflecta at the moment as the naming in Jan's
> code seems to suggest it.

I wouldn't be too happy naming the backend "reflecta". At least for my
slide scanner, Reflecta is just the European distributor for a device
originating from Pacific Image Electronics (PIE).

Just my $0.02 ;-)

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Vleeshouwers, J.M.
2012-09-13 17:06:48 UTC
Permalink
Hi Klaus & Michael,

Too many mails to reply to at once. But it may be practical to choose a name now, before we start merging code.
I don't have a strong preference for 'reflecta'. Since the scanners indicate they are PIE, 'pieusb' might be a more suitable name. Right?

Yours,

Jan

________________________________________
From: Klaus Kaempf [***@suse.de]
Sent: Thursday, September 13, 2012 3:59 PM
To: Michael Rickmann
Cc: sane-devel; Vleeshouwers, J.M.
Subject: Re: [sane-devel] Reflecta ProScan / Crystalscan 7200 PIE film scanner update

* Michael Rickmann <***@gwdg.de> [Sep 12. 2012 07:45]:
> Let us call it reflecta at the moment as the naming in Jan's
> code seems to suggest it.

I wouldn't be too happy naming the backend "reflecta". At least for my
slide scanner, Reflecta is just the European distributor for a device
originating from Pacific Image Electronics (PIE).

Just my $0.02 ;-)

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Klaus Kaempf
2012-09-13 20:22:56 UTC
Permalink
* Vleeshouwers, J.M. <***@tue.nl> [Sep 13. 2012 19:06]:
> Hi Klaus & Michael,
>
> Too many mails to reply to at once. But it may be practical to choose a name now, before we start merging code.
> I don't have a strong preference for 'reflecta'. Since the scanners indicate they are PIE, 'pieusb' might be a more suitable name. Right?

+1

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Klaus Kaempf
2013-03-04 09:02:34 UTC
Permalink
Jan, Michael,

after (too) many weeks, I finally found some time to work on the
'pieusb' backend driver again.

Based on Jan's work, I started a big round of cleanups and coding
style improvements. The various pieusb_*.c are now separate
compilation units.

See https://github.com/kkaempf/sane-backends/tree/kkaempf for the
current state.

The current code compiles but wasn't tested against the actual scanner
yet.
Once this test is successful, I'd like to push the code upstream and
go from there.


Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Paul Menzel
2013-03-04 09:33:45 UTC
Permalink
Dear Klaus,


Am Montag, den 04.03.2013, 10:02 +0100 schrieb Klaus Kaempf:

> after (too) many weeks, I finally found some time to work on the
> 'pieusb' backend driver again.

nice. Thank you for taking the time.

> Based on Jan's work, I started a big round of cleanups and coding
> style improvements. The various pieusb_*.c are now separate
> compilation units.
>
> See https://github.com/kkaempf/sane-backends/tree/kkaempf for the
> current state.
>
> The current code compiles but wasn't tested against the actual scanner
> yet.
> Once this test is successful, I'd like to push the code upstream and
> go from there.

In my opinion, before committing this upstream you could squash several
commits into the original one (as it is not upstream yet). That will
keep the commit log cleaner.

If you have even more time and as you found some small errors already
and fixed them, you could run Cppcheck and splint on your code (and
mention that in the commit message).


Thanks,

Paul
Klaus Kaempf
2013-03-08 16:10:42 UTC
Permalink
Hi Paul,

* Paul Menzel <***@users.sourceforge.net> [Mar 04. 2013 10:34]:
>
> In my opinion, before committing this upstream you could squash several
> commits into the original one (as it is not upstream yet). That will
> keep the commit log cleaner.

sure, no problem. What's the usual process for getting code upstream ?
Are there any coding style guidelines for SANE ?

>
> If you have even more time and as you found some small errors already
> and fixed them, you could run Cppcheck and splint on your code (and
> mention that in the commit message).

My primary goal, besides getting running code, is to make the code
clean and maintanable. These tools will certainly help, thanks !

Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
Vleeshouwers, J.M.
2013-03-11 10:18:04 UTC
Permalink
Hi Klaus,

Thanks for picking this up again.
You are getting into a process with which I am not familiar at all. But I suppose there will be need for some testing - just let me know whenever that is necessary.
I also remember we still need permission to use a couple of algorithms in the IR-cleanup section.

Yours,

Jan

-----Original Message-----
From: Klaus Kaempf [mailto:***@suse.de]
Sent: Friday, March 08, 2013 5:11 PM
To: Paul Menzel
Cc: sane-***@lists.alioth.debian.org; Michael Rickmann; Vleeshouwers, J.M.
Subject: Re: [sane-devel] Reflecta ProScan / Crystalscan 7200 PIE film scanner update

Hi Paul,

* Paul Menzel <***@users.sourceforge.net> [Mar 04. 2013 10:34]:
>
> In my opinion, before committing this upstream you could squash
> several commits into the original one (as it is not upstream yet).
> That will keep the commit log cleaner.

sure, no problem. What's the usual process for getting code upstream ?
Are there any coding style guidelines for SANE ?

>
> If you have even more time and as you found some small errors already
> and fixed them, you could run Cppcheck and splint on your code (and
> mention that in the commit message).

My primary goal, besides getting running code, is to make the code clean and maintanable. These tools will certainly help, thanks !

Regards,

Klaus
--
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfel
Paul Menzel
2013-03-11 14:16:30 UTC
Permalink
Dear Klaus,


Am Freitag, den 08.03.2013, 17:10 +0100 schrieb Klaus Kaempf:

> * Paul Menzel <***@users.sourceforge.net> [Mar 04. 2013 10:34]:
> >
> > In my opinion, before committing this upstream you could squash several
> > commits into the original one (as it is not upstream yet). That will
> > keep the commit log cleaner.
>
> sure, no problem. What's the usual process for getting code upstream ?

I am also new to SANE. But sending your patches to the list for review
and providing a branch to pull from is good way I think.

> Are there any coding style guidelines for SANE ?

They are not documented I think. At least I have not found them. You can
only look at already existing files.

> > If you have even more time and as you found some small errors already
> > and fixed them, you could run Cppcheck and splint on your code (and
> > mention that in the commit message).
>
> My primary goal, besides getting running code, is to make the code
> clean and maintanable. These tools will certainly help, thanks !

Awesome!


Thanks,

Paul
Rolf Bensch
2013-03-12 13:06:27 UTC
Permalink
Hi,

Am 11.03.2013 15:16, schrieb Paul Menzel:
> Dear Klaus,
>
>
> Am Freitag, den 08.03.2013, 17:10 +0100 schrieb Klaus Kaempf:
>
>> * Paul Menzel <***@users.sourceforge.net> [Mar 04. 2013 10:34]:
>>> In my opinion, before committing this upstream you could squash several
>>> commits into the original one (as it is not upstream yet). That will
>>> keep the commit log cleaner.
>> sure, no problem. What's the usual process for getting code upstream ?
> I am also new to SANE. But sending your patches to the list for review
> and providing a branch to pull from is good way I think.
>
>> Are there any coding style guidelines for SANE ?
> They are not documented I think. At least I have not found them. You can
> only look at already existing files.

Please have a look here: http://www.sane-project.org/backend-writing.txt.

I found this link here: http://www.sane-project.org/contrib.html.

>
>>> If you have even more time and as you found some small errors already
>>> and fixed them, you could run Cppcheck and splint on your code (and
>>> mention that in the commit message).
>> My primary goal, besides getting running code, is to make the code
>> clean and maintanable. These tools will certainly help, thanks !
> Awesome!
>
>
> Thanks,
>
> Paul

Cheers,
Rolf
Loading...