CoastColour Blog

Month 36.

In the month of December, the reprocessed L1P and L2R products were made available to the users by http and ftp. The activities were also marked by the finalization of the Round Robin Workpackage, and the preparation of the UCM-4, which will take place on May 9 and 10 in Darmstadt. The in-water product quality is still subject to optimization. Validation of L2R data was continued in December.

Month 35.

In October, the CC processing software had been updated in order to fix the quality issues that had previously been detected (see also October MPR). The activities in the month of November were marked by the validation of the latest products, which showed very satisfactory results over the MERMAID stations, and, secondly, the reprocessing of the complete MERIS FR dataset to L1P and L2R. The in-water product quality is still subject to validation and optimization. Therefore, the production of L2W data will take place in December.

Month 34.

The quality issues which have been discovered in the data during September have been fixed and reprocessing of the validation dataset has been conducted. The data is being carefully validated, showing good quantitative results over the MERMAID stations, and significantly reduced noise over non turbid waters. The in-water product quality is still subject to validation and optimisation. The reprocessed L1P and L2R data will appear in November, whereas the L2W data will be ready in December.

Month 33.

In September the final processing software has been performed. The processing of all CoastColour products is completed and validation by CC partners and users has been initiated by processing of the validation data set. However, some quality issues have been discovered in the data. It has been decided to fix the issues and to reprocess the dataset.

Month 32.

In August the final processing software has been completed and tested. The processing of the final CoastColour products is ongoing and validation by CC partners and users has been initiated by processing of the validation data set.

Month 31.

In July the algorithms have been finalised and further validated. The final processing software has been completed and tested. The processing of the final CoastColour products is ongoing and validation by CC partners and users has been initiated by processing of the validation data set.

Month 30.

In June the main activity focused on the finalisation of the CoastColour atmospheric correction (AC) network. This is now based on a new atmospheric model (R. Santer and F. Zagolski) and uses a dataset with over 6 mio simulations, designed for all types of water. These new L2R and L2W nets were integrated in the software and an comparison between different AC algorithms shows excellent performance of the CC AC over the North Sea and the Rio La Plata as high turbidity water. The so far processed CC L2 dataset is written on tape and available for users on requests. In the Round Robin exercise further results of different algorithms have been received and validated in the harmonized way to include them in the final report. CoastColour has been and will be presented at various events. A manuscript, acknowledging CoastColour, has been submitted to Remote Sensing Environment.

Month 29.

In May several coordination actions (TC, meetings) took place to review the work progress towards the end of the project. The final documentation of all project activities has been prepared. A new data set for the training of a special set of NNs was generated for all sites with extremely high suspended matter concentrations. Also a new version of the atmospheric model (provided by R. Santer and F. Zagolski) was tested and modified to include temperature and salinity effects. 1.5 Mio. cases were simulated and a new set of neural networks was trained. As alternative the atmospheric correction of l2gen was implemented. A validation of different ACs was performed. l2gen was proven to be a good fall-back solution. A visual validation of the final cloud screening shows good results for the new mixed pixels (MIXEL). The snow_ice flag has problems in distinction of sea ice from clouds and the cloud_shadow still needs to be improved. An approach has been developed to introduce ICOL indicators to evaluate the potential impact of the adjacency effects. Due to the end of the ENVISAT mission the NRT service is stopped. Previous NRT data have been moved and can be downloaded from the archive. As part of the Round Robin, a study on the effects of the natural variability of the SIOPs on the retrieval of KdPAR from various algorithms was realized. New results from 2 Algorithm Providers on the matchup datasets were gathered. The kick-off meeting of the IOCCG-CWAC (Coastal Water Algorithm Comparison) working group is being prepared by K. Ruddick. CoastColour was presented at various workshops and conferences, including the Liege colloquium and abstracts have been submitted to different upcoming events.

Month 28.

In April a new versions of the AC and in-water Neural Nets have been tested and discussed during a TC, held on 19th of April. The Nets use ICOL and the Monte-Carlo AC and are based on a previous optical water type classification. The Nets were validated using coastal MERMAID stations and were not improving the results compared to the previous ones. Problems have been identified and are being solved. SeaDAS and l2gen are being tested as alternative AC. The CC MERCI has further been updated. All child products have been reprocessed to CC L1P. The flagging of the reprocessed L1P products have been validated and are found to perform good. Due to a communication problem with ENVISAT the NRT service is currently not available. All NRT data processed until now are available via the CC archive. In the Round Robin the Algorithm Providers are asked to run their algorithm on the matchups dataset. The Round Robin datasets are also used for studying the effect of IOP variability on Kd and Z_eu. The project has been extended until the end of June.

Month 27.

In March Progress Meeting 5 was held in ERSIN to review the status and the further planning of project activities. The progress in March involves the establishment of a new NN approach which includes a prior water type classification according to G. Moore. The L1P processing chain has been updated e.g. with a consolidated geochildgen, a stich tool and a mixed pixel (Mixel) identification. The MEGS processing of all data for 2005-2010 has been finished. A validation of demo CC products over Morocco site has been realised. For general validation purpose the PixBox has been completed with pixels over coastal areas. Results of the Round Robin were presented and discussed at the PM5. The project has been represented at different meetings and workshops.

Month 26.

In March a bases for the validation of the final products has been further prepared. 5600 MERIS FR pixels have been selected for the PixBox to validate flagging. In BEAM a GPF feature has been implemented for the processing and validation of single pixels. MEGS processing of 2008-2010 data is ongoing.
In the Round Robin exercise a draft of the report on algorithm comparison has been sent to the algorithm provider (AP) for review. Some APs provided new validation results to MUMM. Furthermore the IOCCG working group on "Intercomparison of Retrieval Algorithms for Coastal Waters" as follow up to the CCRR is in preparation.

Month 25.

In January the CoastColour team developed further the forward NN, which includes an atmospheric and a water part. Simulations with Hydrolight to produce the second version of the Round Robin simulated dataset are running. In the software developement a polygon merging has been iplemented in the GeoChildGen tool to avoid production of dublicated products. The processing of 2010 data with the MEGS processor has been started. The copying of the L1P and prototype regional product dataset on tape has been started and is still going on. Also some tapes have been written comprising requested data for CSIRO. The Newsletter issue 4 informs of the latest progress in the project.

Month 24.

In December a short report about the UCM3 has been published in the INPRINT 2001/3. The minutes of the meeting have been sent out to the team for review. Also the CoastColour Newsletter issue 4 has been sent out for a review.
A software module for batch processing of FRS products with ODESA has been developed and the software for child product generation has been improved. 2010 data are currently being processed with AMORGOS, MEGS and standard CC L1P, L2R, and L2W. Additionally special requests are being processed for different users with the current CC algorithm. The NRT service is running stable. Data for Africa and Europe are now being delivered to the DDS station at BC.
In the Round Robin exercise a proposal for the simulated dataset version 2 has been sent to the Algorithm providers and a first draft of the Round Robin report is being written.

Month 23.

After the successful completion of User Consultation Meeting 3 the findings of the workshop have been translated into actions. On the algorithm side the QAA algorithm was used for processing and analysis of more data. The NN algorithm was further improved and run on the Round Robin dataset.
The UCM and in particular the launch of the NRT service have significantly increased the product requests from users. This includes requests to get access to all archived data, to the NRT production but also special requests for selected products in order to work closely with the CC team on further product validation.
The second version of the Round Robin Dataset is under preparation.
Technical improvements have been implemented on the production line in order to speed up the NRT processing.

Month 22.

In October the 3rd User Consultation Meeting was the focus of the project activities. 68 participants from 22 countries attended the meeting, which was held in Lisbon, Portugal, from 19.-20. October 2011. The CC team gave an overview of the advances reached during the past year. Several Users took the opportunity to present and discuss the results of their validation of CoastColour products. After the main meeting 13 Algorithm Provider participated in a Round Robin Workshop to discuss algorithm performance. In parallel a CoastColour Products Interactive Workshop and Training Course was given to a group of 20 Users, and included guided hands-on exercises on CoastColour products using the BEAM software. The NRT service was started during the UCM3 and provides CC L1P, L2R and L2W data 24 hours after acquisition time latest. Access to NRT and also to the complete CC archive is open.

Month 21.

In September the bio-optical model of the new neural nets have been improved by training a separate net for each of the 4 water constituents phytoplankton, detritus, yellow substance and mineralic suspended matter. For the processing of CC L2 products a regionalised aerosol model for atmospheric correction has been implemented. A processing chain has been defined to compare and validate L2 products from MEGS8.0, ODESA and CC L2R, with and without using ICOL at L1. The correction of the sky dome reflectance has been described in a Technical Note.
In the Round Robin Results have been received from Algorithm Providers and are being validated by plotting in a harmonised format. Work on the general CC validation dataset and special datasets requested by users for validation purpose is advancing.
The Near Real Time Service is being set-up. Download of products from the ESA roling archives is establised and the CoastColour L1P, L2R and L2W is being connected. The NRT service will be launched at UCM3.
User Consultation Meeting UCM3 preparation has entered into its final stage. More than 50 participants already registered.The agenda with abstracts is now available on the CC website.

Month 20.

In August LISE has driven forward the regionalisation of the Atmospheric Correction (AC). The CC AC neural network is now based on this new aerosol optical model. The potential has been investigated, to use forward neural networks in combination with an optimisation procedure and the computation of uncertainties for the AC and in-water inversion. In Round Robin results of algorithms applied to the various CCRR datasets by different users have been uploaded to the Round Robin ftp server. An updated version (v1.1) of the CCRR in-situ reflectance dataset has also been uploaded. In addition MUMM is extending the simulated data with input SIOPs of version v1.0 to include Kd and Z_eu.

Month 19.

The processing of L1P of all years 2005 - 2009 was completed, and the data are now publicly available. The input data of the year 2010 have been shipped from ESA to BC. On 13. and 14. July Progress Meeting 4b was held in Hamburg. The Prototype products and the L1P data were reviewed by the team, and the next steps (UCM3, final processing) were discussed. Algorithm development work was done for experimental FLH and Primary Production. Validation work concentrated on atmospheric correction and assessing the impact of adjacency effect. A video conference was hold on 20. July to discuss the status of Round Robin task with the participants.

Month 18.

Activities in June concentrated on provision of the L1P dataset to users and the demonstration dataset to Round Robin participants. Further improvments of L1P processing have been achieved and implemented. All data available for CC testsite from 2005 to 2009 have been processed to Level 1P. Upload to ftp server will be finished in the first week of July. For higher level products 3 different Neural Nets have been trained and implemented. The demonstration Dataset of regionalised L2R and L2W products of 2006, including match-ups for validation, have been processed and are available to the Champion Users. Experimental novel FLH algorithm and PPP algorithms have been developed and applied on selected test sites. Processing and first validations of the products are described in different document deliverables.

Month 17.

In May last preparations for the User Consultation Meeting 3 in Lisbon have been realised. The meeting is open for registration and abstract submission now. Processing with AMORGOS was successfully completed and focus shifted to preparation of the demonstration data set. After major improvements, escpecially for pixel classification, Level 1P processing started. The training of different Neuronal Nets for regionalisation was partly successful but improvements still have to be achieved.
The Demonstration Dataset (1 year of MERIS L1P, L2R, L2W) is available for Champion Users, and for general public users L1P data are available. We are currently upgrading the Website to distribute the data; until available, please send an Email to request access to the ftp account.

Month 16.

The major milestones in April 2011 were the first regionalised algorithms, using the CoastColour in-situ dataset and the NOMAD database, and the Round Robin algorithm provider meeting. These two events mark the moment when regional Case2 algorithm are applied to CoastColour MERIS L1P data. The processing of the data continued at full speed.

Month 15.

The month of March 2011 was an intense working period concentrating on three lines of activities:

1. the preparation of the in-situ data, made available by the users, has been completed. The resulting consolidated dataset was carefully analysed by the CoastColour team, in order to identify the records to be used for the Round Robin, and to identify regions where sufficient data for a regionalisation of the algorithms is possible. This activity has been concluded.
2. the finalisation of the Round Robin dataset by extracting match-ups corresponding to the in-situ data and
3. to startt calibrating regional algorithms, including a new global algorithm based on NOMAD data.

Month 14.

In February progress concentrated on completion of processing geo-childgen subsets with AMORGOS, and the final organisation of in situ data available for generic algorithm developement and analysis of their practicability for regionalisation. The first generic algorithm has been trained. Preparation of matchup extraction with PixEx tool has been successfully completed and extraction is started. In Round Robin a beta version of in situ reflectances dataset was provided. A proposal for an IOCCG working group for extending CC Round Robin activities has been presented to IOCCG.

Month 13.

In the new year the work was commenced with the CoastColour water processing algorithms. In order to generate regional models and subsequently regional parameterisation of the neural network and/or tuning parameters for the QAA, large number of radiative transfer calculations with Hydrolight have to be performed. This work was undertaken by porting the tool to a large Cluster at HZG. A second strand of activities was the consolidation of in-situ data, which was completed, and the continuation of the Round Robin excercise. Thanks to the consolidated in-situ database the reflectance dataset of the Round Robin Datapackage was completed.

Month 12.

December was dedicated to work on the CoastColour algorithms, procesisng and consolidation of in-situ data. The radiometric processing tool was completed and generation of auxiliary data continued. The atmospheric correction was completed. The work on the update of the water processing was initiated. A new version of Hydrolight was implemented and extended for temperature and salinity models. A study was performed for new fluorescence line algorithms. The processing of the Level 1 products was continued.

Month 11.

November 16.-17. The first User Consultation Meeting was hold in ESRIN. More than 70 participants attended this meeting. The CoastColour team presented the status and and further planning of the project, and gave demonstration of CoastColour products. User presentations gave insight into the coastal work which will benefit from CoastColour product. The meeting was concluded with a number of recommendations, and it was agreed to make CoastColour products in an early stage available to users for an interactive review and improvement. Users and CoastColour team appreciate this commitment of users to actively participate. During the meeting the release of the CoastColour Round Robin Data Package was announced. The data package can be downloaded from the CoastColour Website.

Month 10.

During October 2010 many users provided in-situ data from their site to the CoastColour team. This is an important input for regional algorithm calibration, and for compilation of the Round Robin Data Package. While the provision of concentrations of water constituents is well progressing, only few measurements of water radiometry are available; however, these are important for the Round Robin and the product validation. The Round Robin Protocol has been published and the compilation of the Data Package is progressing. The L1P processing and the L2 algorithm development are well underway. A major milestone has been achieved by implementing a 2nd to 3rd reprocessing calibration correction. CoastColour was presented at several occasions, including amongst others SPIE Asia-Pacific Remote Sensing conference held in Inchoen, South Korea on 13th October 2010.

Month 9: Progress Meeting 2 and Preparation of the Round Robin.

September 2010 started with the second progress meeting in Plymouth, at PML. It was attended by all project core team members and the consultants Lee (MSU, USA), Schroeder (CSIRO, Australia). The Science Team was represented by S. Peters (IVM, Netherlands). The meeting was held in conjunction with Kick-off meeting of the Ocean Colour Climate Change Initiative.
During the Progress Meeting the L1P and L2 algorithms were intensively discussed; new ideas to handle bottom reflection were taken onboard following a method presented by Lee. A critical technical issue adressed the definition of the geometry in MERIS products, which is potentially responsible for significant errors in the atmospheric correction in eastern parts of MERIS images currently (in Standard and CoastColour processing). However, this was discussed controversely at the meeting.
The following weeks of the month were intensively used to implement and test the new atmospheric correction, and the preparation of the Round Robin exercise: the protocol was finalised and published on the CoastColour Website. The work to compile the four datasets comprising the Round Robin Datasets was started.
CoastColour was presented by Z. Lee at the Ocean Optics conference in Anchorage, Alaska.

Months 7 and 8: In July and August the work on the CoastColour algorithm proceeded.

The equalisation algorithm was implemented and tested; it removes largely the coherent noise in the Level 1 products. The atmospheric correction was significantly progressed by compiling the regional aeorsol database, and developing the alternative atmospheric correction on the basis of the ICOL (Adjacency correction) procedure. A critical issue has been identified concerning the definition and usage of the viewing and sun angles.
If used correctly the water leaving radiance is significantly changed, and validation proves the positive impact, e.g. on the chorophyll product. Experiments were carried out to study carbon and primary production algorithms, and the IOP model for North Sea site was improved.
In parallel the user interactions were further intensified. New users were contacted and a champion user from South America joined the team.
Presentation of CoastColour at major conferences in autumn and publications were prepared. The User Consultation Meeting (Nov. 2010 in ESRIN) was announced on the CoastColour Website, including the possibility for users to submit an abstract for an oral presentation.

Month 6: Level 1P processing, Level 2 Algorithm development and Presentation at ESA Living Planet Symposium

The work in June was characterised by the further processing of Level 1P products, and the continuation of the development of the Level 2 algorithms.
Progress was made on the atmospheric correction, where a database of regional aerosol models was compiled, and where a method to identify and correct cocolithophores was developed and tested.
The QAA processor was transferred to the BC code repository, and BC team together with Dr. Lee started on the update of the software to be implemented in the CoastColour prototype. Using the current version of the CoastColour processor, sample products were generated for each CoastColour site.
Images were generated and put on the public Website together with the sample products.
The update of the Website is currently under review and will be made public during July.

Month 5: PM1 and Requirements review Meeting, Level 1P processing and L2 algorithms

Month of May started with the Requirements Baseline Review Meeting / Progress Meeting 1, which took place on 4. and 5.May in Loicz Office in Geesthacht. The presentation by the LOICZ CEO Hartwig Kremer set the frame for this meeting, by highlighting the importance of water quality observations from space for the coastal zone.
Right after the meeting the list of sites to be addressed by CoastColour was consolidated; The list has been enlarged compared to the original one by new sites from Africa, South America, Asia and the Arctic. This list maybe extended further later, but the processing has then been started for this set.

The rest of the month was characterised by intense work on processing of MERIS FRS data over these sites, collection of in-situ data and development of the Level 2 algorithms.
Interactions with users was continued.

Month 4: Completion of User Requirements Analysis, Data ingestion and Level 1P processing

This month the user requirements analysis was completed. The user interviews were completed, and by the end of this month each of the 32 champion users, having sent a user questionnaire in 2009, was contacted by one of the CoastColour parterns.
Important feedback was gathered during this process, e.g. concerning data availability and product requirements, and the availability of in-situ data by the users was investigated. Here the situation is better than assumed; quite some users have suitable data for regional algorithm adaptation, and most users have data suitable for product validation. Access to the data was discussed, and this will remain a critical issue to be further adressed.
As one action, a CoastColour Data Policy was drafted. Further, several follow-up activities were considered, including personal visits or possibilities to gather dedicated in-situ data during ship cruises.

4 new users joined the CoastColour user community. Though this the global coverage of CoastColour was significantly expanded and includes more African and Asian coastal waters. Discussions with users from the west coast of South America have been started.

The ingestion of screening of MERIS FRS data was continued. The processing steps for the Level 1P processing were specified as start of the work for the Technical Specification.

Month 3: User Consultation and Requirements Analysis, initial ingestion and processing of MERIS FRS L1 products

The first strand of acitivities in March was the review of the user requirements and consulation of users. All users were informed about the CoastColour project and the process to contact them personally was continued. The product requirements and the data available for algorithm calibration and validation were discussed. The second strand were the set-up of a CoastColour MERCI system, where all delivered MERIS FRS products will be ingested, registered, quality checked and quicklooks will be generated. This process was completed for the complete dataset of the year 2005. The third strand was the work on the both the Level 1 (geometric improvements, adjacency effect) and on Level 2 (water model, atmospheric correction). In parallel to this technical work the project was presented at the UK GMES conference, and events such as the Lining Planet Symposium or the UCM-1 were prepared. The Website and the project Wiki were updated with additional information.

Month 2: Kick-off Meeting and User Requirements Analysis

The kick-off meeting was held on 4. and 5. February in ESRIN. It was attended by key users from Europe, Australia and Africa, ESA members from various groups interested in coastal remote sensing, all members of the instrustrial/scientific team (Brockmann Consult, GKSS Research Centre, Plymouth Marine Laboratory, MUMM, LISE ADRINORD, Institute for Oceanography University Lisbon) and members of the consultants team. The meeting was held in a very postive and constructive atmosphere and demonstrated the large committment of all contributing parties to the project. During the course of the month of February the product requirements were reviewed and the possbible algorithmic solution were discussed. The CoastColour Website was launched and project was presented and discussed at the SAFARI meeting on 15.-17.02. in Kochi/India, and subsequently with the members of Chlorogin.

Month 1: Start of the project and initial activities

The official start of the CoastColour project is the 04. January 2010. The project shall be completed on 31.12.2011. The first month of the project was characterised by activities to getting the project started: preparation of sub-contracts, staff recruitment, setting up of the internal project management facilities (project Wiki), design of the CoastColour logo etc. Secondly the review of the user requirememnts started. The complete list of user requirement questionnairs was reviewed. An revised list of products and algorithms was drafted. This will be further discussed with the users in order to conclude with a final list in the user requirements document. The 2 days Kick-off meeting, 4.-5.2.2010 in ESRIN, was prepared.