Public Lab Research note


GSoC proposal: Mapknitter ORB Descriptor (w/ auto-stitching, pattern training, and live video support) and LDI revamp (major UI enhancements)

by rexagod new contributor | | 172 views | 20 comments |

Read more: i.publiclab.org/n/18515


About me

Name : Pranshu Srivastava

Email : rexagod@gmail.com

Github : https://github.com/rexagod

Proposal: Includes MapKnitter UI and Microscope live stitching, auto-stitch in MapKnitter (magnetic attraction))

Student year: Sophomore (second-year student)

Field of study: Computer Science Engineering, B.Tech.

Affiliation : Jaypee Institute of Information Technology, India

Location : Noida, India

Project description

Abstract

Employing machine learning techniques to incorporate an ORB desciptor (w/ auto-stitching, live video support) in order to enable better pattern matching along with some major LDI UI (and API) enhancements.

Aims to solve

Both of the projects listed below are detailed descriptively in the implementations section

Oriented FAST and Rotated BRIEF (ORB) Descriptor)

  • Currently, the user has to manually "stitch" similar looking images in order to form a larger one. We aim to automate this cumbersome process in the following sequence.
    • First, using some of the advanced machine learning techniques, mostly "Oriented FAST and Rotated BRIEF", i.e., ORB detection, offered by the JSFeat (JavaScript Computer Vision) Library the algorithm will understand how similiar any two images passed to it are, showing the corresponding outputs (keypoints, matches, good matches, gaussian blur and grayscale intensity) for each pair to the user. Also, a "training process" will be initiated every x seconds to better recognize and discover even more keypoints in the image, leaving lesser room for failed attempts at each iteration.
    • Second, on the basis of the above details the user can choose to "stitch" the images which itself, will be an automated process, and the "auto-stitcher" ToolbarAction will proceed to automatically stitch them together (using the Auto-Stitcher module, detailed below).
    • Lastly, for better accuracy, the user themselves can intervene and make small adjustments if they wish to obtain precise results.
  • Public Lab will also showcase a live video ORB matcher, that learns from an image passed to it, and goes on to find that particular pattern in the live video feed! At each step, the most efficient params will be considered, so as to maintain the maximum render FPS possible. This can be applied to various areas of interest, for eg., a particular minute organism can easily be detected if the video feed from the microscope was supplied to this module and compared against an image of it (downloaded from the net, or uploaded from user's local), or a prominent object in space using the satellite's live feed, the possibilities are endless!

Mapknitter UI enhancements

Currently, the Leaflet.DistortableImage repository serves as the supporting image distortion library of the Mapknitter repository hence proving itself to be an essential part of it. We'll be implementing some major core changes as per the planning issue describes here, which will definitely prove beneficial to the user in terms of better accessibility and interactivity. Work regarding this already in progress and can be checked out here.

Implementation

Oriented FAST and Rotated BRIEF (ORB) Descriptor)

Dividing this into subparts,

a) Microscope live-tracking: We can modify the algorithm to adapt to just a single image and train only on that rather than depending on the live video feed to train itself, in short, this will allow the user to provide an image (pattern) as an input and the tracker will then attempt to find that pattern in the feed every x seconds in a loop (currently set to 2s). On finding appropriate matches the tracker will filter the number of "good matches" and if they are above a certain threshold, display "matched".

Screenshot from 2019-03-09 19-36-13

Live-tracking in action! (do let me know if the .gif is taking way too long to load)

Peek-2019-03-09-18-10.gif Few points of interest above that one should notice are that the no matter the orientation, ORB will recognize image pattern if they are even partially visible. Also, notice how it confirms a match case even when some portion of the coaster was cut off.

b) ORB Algorithm (currently in beta, click here to check it out!): We can enhance the current ORB algorithm in the following ways in order to better fit Public Lab's utilization.

  • Completely remove all the video instances since we won't be working with any video formats whatsoever inside the LDI UI.
  • Add support for training process to occur b/w images rather than a video and image.
  • Calibrations and adjustments below can be employed to maximize ORB coverage while reducing outliers at the same time. But what exactly are these and why do we need them?
    • Gaussian blurring: Reduces noise (outliers) by blurring the modified input img_u8(8-bit conversion). Screenshot from 2019-03-09 20-18-41
    • Grayscale method: Converting every pixel to a 0 (black) or 1 (white) value for way better performace and evaluation. Screenshot from 2019-03-09 20-31-14
    • Match threshold application: Specify matching standards (how high/low should the ORB expect the match instensity to be in order for it to be a good match). Screenshot from 2019-03-09 20-37-39

The idea is to separate the image into two parts; the background and foreground. Select initial threshold value, typically the mean 8-bit value of the original image. Divide the original image into two portions; Pixel values that are less than or equal to the threshold; background.

Based on this, we need to consider two more thresholds, which are available under the YAPE06 module, originally designed by CVLab, Switzerland.

  • Eigen threshold application: Specify how similar two pixels should be in order for the ORB to perceive them as "lying in the same cluster". Screenshot from 2019-03-09 23-58-04
  • Laplacian threshold application: Defines the rate of change of intensity of the pixels that should be perceived by the ORB as "noisy". Screenshot from 2019-03-09 23-55-57
  • Finally, we can modularize this technique into a custom PL module that accepts two images and runs the algorithm against them. Refer (d) to look over the current proposed steps we can take after this.

c) Auto-Stitcher module: The auto-stitcher module will accept two images, the two most recently clicked images (as of now) and then then proceed to "stitch" those together as per the instructions passed on to it. We can, as an initial implementation, add a functionality to this module that allows the user to attach an image in any of the four major directions, after which the user themselves can make little changes to those images if need be. Since it wasn't merged at the time of writing this, I didn't implement this using the "multiple-image" selection feature, but am in the favor of shifting from the "last two latest-clicked images" to selecting any two images as per user's convenience and passing those down to the Stitcher, thus improving the module's overall accessibility.

// an instance that "traces" the last two latest-clicked images
var Tracer = L.DistortableImage.Tracer = L.DistortableImage.Tracer || {};
initialize: ...
//previous image (constructing param being {}) is available in this scope
overlay.Tracer = Tracer;
Tracer = overlay._corners;
//current image that is clicked is available in this scope

carbon

autostitcher--road.gif

d) Integrating (b) and (c): We can incorporate the ORB and Stitcher modules to work together in real-time in the MK UI. Whenever the user selects two images for comparison, the images are first passed down to the ORB module that displays all the relevant information about that particular pair of images in a modal. After this the user can select whatever pair fits best and choose to "stitch" them together in a user-defined orientation (right, left, top, bottom) with respect to the original image using the Auto-Stitcher.

modal

Mapknitter UI enhancements

This is currently under progress, and most of the points in the projects and compatibility section have been or are getting resolved over in the PRs which I've submitted till now. I do aim to incorporate the appropriate modifications to my remaining open PRs as per @warren's suggestions and get them merged speedily and the remaining ones as well. You can checkout the follow-ups on the original planning issue as well.

Timeline

  • 27th May to 23rd June: Introduce and integrate two new PL modules (w/ blog posts on both dates)
    • Week 1 and 2: ORB and Auto-Stitcher modules (w/ FTOs on each weekend)
      • Starting off with the most exciting aspects of my proposal, the ORB and Auto-Stitcher modules will be implemented from scratch and solidified along the way (details above), while keeping in mind to make these effective yet simplistic, so that newer contributors who find this interesting can easily contribute to this. I will also be fully documenting both of these modules to make the code as readable and easily understandable as possible, especially the Oriented FAST and Rotated BRIEF (ORB) module. That being said, I, under the guidance of my mentor, will execute appropriate measures to refine these modules so that they perfectly fit in with Public Lab's coding standards, practices, and community expectations.
    • Week 3 and 4: Integrate modularized ORB+AS and Microscope web-based live demo (w/ PR reviews on each weekend)
      • Although I've proposed a way to integrate ORB+AS into LDI above, I am and will be open to newer and better ideas that might be suggested by my mentor during my GSoC period. As is discernible, this will be a more work-focused period, and throughout this duration I will work on a modularized integration in order to maintain the readability of the codebase.
      • The Microscope web-based demo will implement an abstract flavor of the ORB different from its native or LDI implementation. For this purpose, I'll be customizing the ORB almost from the bottom-up, to change its current behavior of depending on user to train itself whenever prompted from the video feed to completely discarding that approach and rather train itself from a specific image supplied to it, so that it could find the desired patterns in the live feed, as is demonstrated the .gif above. It may also be worth noting that this will be a PL standalone module, unless until suggested otherwise.
    • 24th June to 28th June: Evaluations
  • 29th July to 21st July: Testing and Debugging Phase (w/ blog posts on both dates)
    • Week 1 and 2: General Debugging at LDI's #87 (w/ FTOs on each weekend)
      • This period will consist of different approaches to resolve the issues (todos, lower priority, and leftovers) over at LDI #87. These bugs have been in the system for a while now, and need to eradicated as soon as possible to build a neat codebase to attract more contributions in the future and less dangling errors in the future.
    • Week 3: Unit Tests w/ Jasmine (w/ PR reviews on each weekend)
      • Initially proposing the opportunity to write extensive tests for the ORB, AS, and the Microscopic live-feed modules, but willing to write exhaustive unit tests for different potential LDI modular fragments, however my mentor sees fit.
  • 22nd July to 26th July: Evaluations
  • 27th July to 25th August: LDI UI revamp (w/ blog posts on both dates)
    • Week 1 and 2: LDI #126 (w/ FTOs on each weekend)
      • During this period I will be looking forward to completely solving the remaining planning issues, i.e., finishing off the projects section and implementing the assorted utility functions, a step closer to the next major release of LDI!
    • Week 3 and 4: Uncommon issues (w/ PR reviews on each weekend)
      • I will finish off my GSoC journey by working on some of the uncommon issues such as Cross-browser testing for the Toolbar module by generating heap snapshots across various combinations of OS and browser versions. I'm currently considering different ways to implement this but this method looks the most promising and solid. Other than this, I will work on implementing a couple of @sarahelson81/save-time-on-manual-cross-browser-testing-3772be756e68">unique approaches that'll further guarantee the end-to-end functionality of PL's components and will also be easier for the new comers to quickly participate in! Also, I'll also look over the menu placement and image deletion (and specific ordering) issues which have been showing unexpected behaviour for a while now.

Needs

Only the guidance of my mentor, everything else that I require is either online, or on my local. πŸ˜„

Contributions

At the time of writing this down, my contributions have been as follows.

Screenshot from 2019-03-10 02-41-12

  • Pull Requests (53 total, 46 closed, 7 open): Navigate
  • Issues -- FTOs (17 issues): Navigate
  • Issues -- Overall (31 issues): Navigate
  • Comments (on 148 issues): Navigate
  • Review Requests (19 PRs): Navigate

Experience

Comfortable with: C, CPP, JS, SH.

Achievements

  • Mentored at KWoC organized by KOSS, IIT Kharagpur, accessible here (2018).
  • JS Core at GDG JIIT Noida (2019) and JIITODC Noida (2019).
  • Secured 1st rank at DevFest organized by GDG JIIT Noida (Google Developer Groups, now Developer Student Clubs) (2017).
  • Secured 1st rank at Website Designing Competition (WDC) organized by GDG JIIT Noida (2017).

Teamwork

  • Interned towards the end of my first year as a Front-end developer at Profformance Technologies Pvt. Ltd. for their product, Xane, a React chatbot for HR purposes. The team strength wasn't much, about 25 people or so, and after a couple of months, I adapted to a work-from-home option to manage it with my studies, while visiting occasionally to attend meetings and other events. Throughout the duration of my internship here, I was introduced to the corporate etiquettes and productive practices that helped me realize the importance of time and priority management. I employed these techniques into my daily lifestyle to stay motivated and focused towards my project, as is discernable here too.
  • I am a member of JavaScript core team at Google DSC JIIT. I've been teaching JS (Vanilla, React, and Node.js) to my peers for some time now which has helped me to stay updated with the latest API changes as well, while coordinating with rest of the team throughout.
  • I also mentor a lot of enthusiastic students about OSS through JODC, working collaboratively with a lot of local tech societies like DSC-BVP, I take workshops and talks to bridge the gap between newcomers and development and spread awareness for FOSS. I am a teaching assistant here as well.

Passion

Screenshot from 2019-03-10 02-37-11

To start this off, I cannot stress how joyous I am when I realize that my code is actually improving the very surroundings that we live in! It's really an amazing feeling to "give back" to nature in such a unique way, which has become my passion and should be everyone else's too! Having said that, what interested me most about Public Lab's projects was the substantial, caring, and progressive community, which actively helped me clear away my doubts and learn new and interesting things throughout my journey with Public Lab, which I very much plan to extend for the many years to come!

Audience

My project will directly help all users (students, researchers, etc.) working in all kinds of fields that relate to studying, examining and experimenting with maps in many different ways possible, as described in various sections above. Also, it will help them recognize patterns that might often be "overlooked" by the human eye and thus extract credible information from live-video feeds, ranging from microscopic experiments to satellite broadcasts to cost-effective visual sensors for the visually impaired.

The best thing about this is that all of it is open-sourced and free, so people from all over the world, no matter what their current social or financial status is, can use this to meet their needs, and even make it better!

Commitment

I do understand that this requires serious commitment, and since the day I joined Public Lab I've structured every single day to revolve around only two priorities, i.e., PL and college, actually in that order (I like it here!), which has enabled me to make about 185 contributions since mid-December! Also, I'd like to mention that I will, as I have done previously, continue to actively interact with the newer contributors and provide insight and any help that I can regarding their PRs and issues during (and after) my GSoC period. Hence, I firmly believe that I will deliver my assignments with commitment and promptness, given that I get selected!

Thank you!


17 Comments

This is really great, thank you @rexagod!!! A very powerful and interesting proposal.

I'm interested in how the process breaks down into specific functions that are re-usable -- what are the useful subcomponents of this project! Such as - could it have such sub functions as:

  • lib.getPoints(image)
  • lib.findPointMatches(pointsArray)
  • lib.findPosRelativeToImage(imgA, imgB)
  • lib.findPointsInImg(pointsArray, img)

These may not be exactly right, but trying to think like this to give us a set of tools that could be reconfigured in different ways, such as to run on video frames, to turn a video stream into a large image, to move Leaflet image overlays around, things like that.

There might also be listeners, like lib.onFindMatch(fn) which would then run function(matchedPoints) or something... all these would be great too because they'd be testable!

What do you think of breaking down the architecture a little bit like this? Thank you! This is off to a great start!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Awesome start to the proposal πŸ˜ƒ !!!

Reply to this comment...


Hey @warren @sagarpreet! I'm really glad that you liked the proposal! πŸ˜ƒ

Sorry for the delay, the past couple of days have been very tedious due to college stuff! But that's all done with and I've compiled together a list of the functions/data structures used in this project below. @warren, should I add a proper documentation regarding the same in the proposal above, since I thought that I'll be doing that in the code itself in the first couple of weeks? But, I'd be happy to anyway! πŸ‘

Fn./Structure name Description
matrix_t Basic structure to store data in terms of rows/columns, or image width/height along with the respective signatures (usually jsfeat.U8_t | jsfeat.C1_t -- single channel unsigned char)
match_t Records learned variables (screen index, pattern index, pattern level, distance) in a structure
demo_opt Controls user-defined options along with critical training
train_pattern Maintains original aspect ratio, apply grayscale and resample, prepare preview canvas with half (if not overridden) the original dimensions, preallocate corners and descriptors, detect keypoints in a loop as defined by num_train_levels, and finally store their coordinates after adjusting respective to the preview canvas size (determined earlier)
demo_app Perform final steps such as line colors, stroke styles, number of descriptors to define, and displaying info through stat object
tick Major rendering fn. (applies filters, renders patterns and matches back to the canvas, and supply info to the #log element)

Rest are util fns. that majorly make use of the JSFeat APIs to perform primitive objectives as is indicated by their definitions. These are,

  • detect_keypoints: Uses yape06.detect method to perform detections
  • find_transform: Calculate line geometry between keypoints in both canvases and re-estimate the same if motion change is detected in the canvas
  • match_pattern: Performs naive brute-force matching. Each on screen point is compared to all pattern points to find the closest match.
  • render_matches: Draws a match_mask(connecting lines) depending on the params passed down to it.
  • render_pattern_shape: Draws a box around the most probable area, with the highest number of keypoints.
        ctx.lineTo(shape_pts[1].x, shape_pts[1].y);
            ctx.lineTo(shape_pts[2].x, shape_pts[2].y);
            ctx.lineTo(shape_pts[3].x, shape_pts[3].y);
            ctx.lineTo(shape_pts[0].x, shape_pts[0].y);

Also, I do believe that it'll surely be a good practice to abstract off as much methods as we can which will definitely help us in the future with custom functionalities and testing requirements! Thanks!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@warren I'm also currently working on an unorthodox technique that'll help us soon incorporate specific image ordering functionality.

ordering

I'm taking this over to GitHub now, and will soon share updates and discuss my approach regarding the same there! Thanks!

Reply to this comment...


Just going through this and it'll take me a little time, but thank you! As to copying it into the proposal, that's OK, we can see it all in the comments. You're right that it'd eventually hopefully make it into the README and such, so no big deal right now. Thanks!

Reply to this comment...


OK, hi @rexagod - just skimmed this but actually I think there may need to be a lot more separation between the UI (say, drawing boxes, lines, things like that) and the underlying code that drives it. My question about getPoints, findPointMatches and the other methods was to look at what functions would be useful for general-purpose uses, without yet mixing in any display code. I want to know how you'd pass in 2 images, and how we'd format the identified points that were handed back out? How we'd "save" a point (in what format) that we want to reference again later? And how to (most importantly) return the location of a point in Image B but using the coordinate space of Image A. (i.e. what point in Image A does a given matched point in Image B correspond to?)

Finally, maybe as important as that last, can we return the four corner points of Image B in Image A's coordinate space? So that we can position Image B relative to Image A?

Does this make sense? These don't yet get into how the info would be displayed, and I think it's important to develop a strong set of function descriptors before actually writing any display code!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


And... THANK YOU!

Reply to this comment...


@rexagod this is really cool! Really excited to start to see some of these ideas implemented and feel free to make suggestions on my multiple image select PR if you see early opportunities for structuring the code in a way that would help you implement this in the future.

Reply to this comment...


Hey @sashadev-sky! I'm glad you like it! πŸ˜ƒ Apologies for not being able to chime into the multiple image PR, though I've been meaning to for a while now, and definitely will! I've been nonetheless following up on it and it looks very promising! πŸ‘

@warren Please refer the text/images below and do let me know if there's anything else I can help with!

pixels

If we assume the canvas to be say, 640x480, the ORB will run against 307200 pixels (stored in matches array above) against every pixel in the original pattern. Visualize this as the bigger canvas (let's call it screen canvas) having it's bottom left corner at the origin with the measurable units on both axes being multiples of a pixel. Similarly let's call the smaller (inset) canvas pattern canvas. That being said, let's get down to the questions.

I want to know how you'd pass in 2 images, and how we'd format the identified points that were handed back out?

The pattern_xy (matched points in smaller canvas) and screen_xy(corresponding matched points in larger canvas) arrays are globals and thus can be accessed after they have been assigned values from wherever we'd want to.

How we'd "save" a point (in what format) that we want to reference again later?

Similar to above, the points are already saved in the two arrays which can be traversed in a level-order fashion to get all the identified points in the canvas. As far as the format for storing these points is concerned, pattern_xy and screen_xy are object arrays that store the xy coordinates of the identified points. Please refer to the snippet below.

// construct correspondences
            for (var i = 0; i < count; ++i) {
                var m = matches[i];
                var s_kp = screen_corners[m.screen_idx];
                var p_kp = pattern_corners[m.pattern_lev][m.pattern_idx];
                // identified points in the smaller canvas
                pattern_xy[i] = { "x": p_kp.x, "y": p_kp.y };
               // identified points in the larger canvas
                screen_xy[i] = { "x": s_kp.x, "y": s_kp.y };
            }

And how to (most importantly) return the location of a point in Image B but using the coordinate space of Image A. (i.e. what point in Image A does a given matched point in Image B correspond to)?

The xy sets on the left represent matched points in the pattern canvas pattern_xy and those on the right represent the corresponding matched points in the screen canvas screen_xy.

Screenshot from 2019-03-16 21-21-07

If any pixel (on the whole canvas) has a good match data, we send that into those arrays.

if (match_mask.data[i]) {
                        pattern_xy[good_cnt].x = pattern_xy[i].x;
                        pattern_xy[good_cnt].y = pattern_xy[i].y;
                        screen_xy[good_cnt].x = screen_xy[i].x;
                        screen_xy[good_cnt].y = screen_xy[i].y;
                        good_cnt++;
}

Finally adjust the corresponding "scale" for the main canvas.

 // fix the coordinates due to scale level
                    for (i = 0; i < corners_num; ++i) {
                        lev_corners[i].x *= 1. / sc;
                        lev_corners[i].y *= 1. / sc;
                    }

Finally, maybe as important as that last, can we return the four corner points of Image B in Image A's coordinate space? So that we can position Image B relative to Image A?

The shape_pts array specifically stores the coordinates for the given purpose derived from the homo3x3.data, a homography pin-point helper that serves the sole purpose of what its name suggests, and is rendered out as below.

var shape_pts = tCorners(homo3x3.data, pattern_preview.cols * 2, pattern_preview.rows * 2);
            ctx.moveTo(shape_pts[0].x, shape_pts[0].y);
            ctx.lineTo(shape_pts[1].x, shape_pts[1].y);
            ctx.lineTo(shape_pts[2].x, shape_pts[2].y);
            ctx.lineTo(shape_pts[3].x, shape_pts[3].y);
            ctx.lineTo(shape_pts[0].x, shape_pts[0].y);

We can also adjust the scale in the larger canvas (as above) to display the "matched" box in the smaller one too.

Thank you!!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Well detailed and well structured proposal. I really appreciate the depth in which you have discussed the things. Few suggestions We are looking forwards for supportive folks. Make sure to do 1. at least one PR review each week and

make some FTO issues in each week to involve new contributors inside your projects. People will love to be part of big projects. It is a great feeling to help others.

Please take out a day from timeline from each month at end of each phase of SoC fellowship to write about what you learnt and did in the period. Earlier it was not compulsory. But let's make it compulsory. By this mentors will be able to access your progress. Don't forget to mention the ftos which you created during this period. Also mention the PR reviews which you did.

We are planning for video calls I'm each month during SoC program. So, it will be great to see you all.

It will be great if you can review each others proposals and give some suggestions.

Also, it will be great if you want to (optional) to hold a conference at your university and explain Public Lab. What we do here? Programs and our workflow. This will be great help to public lab.

These are small activities which can lead to better and supportive community at PL. All participants all programs are encourage to write these in their proposals.

Thanks and best of luck.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Hey @bansal_sidharth2996! Thank you so much for responding to my proposal. I will surely see to it that these suggestions are followed in the future just as I've done them in the past.

Please take out a day from timeline from each month at end of each phase of SoC fellowship to write about what you learnt and did in the period. Earlier it was not compulsory. But let's make it compulsory. By this mentors will be able to access your progress.

Had this in mind for a while (I've been delaying the idea of having a blog for some time now and this would be a great start! πŸ˜ƒ ). I'm currently considering between a more "exposed" blogging system like Medium or something a bit more "abstract" like Dan's overreacted.io, or maybe I should stick to publishing my monthly progress directly to publiclab.org? What do you think?

Don't forget to mention the ftos which you created during this period. Also mention the PR reviews which you did.

A bit unsure about what period (the current or the SoC one) are you referencing to, and do I mention this in my monthly SoC post or my proposal? My FTOs and PRs (w/ reviews) till now are listed in the "Contributions" section above. Also, as far as the SoC period is corcerned, as is mentioned above, I'll absolutely continue to open up interesting FTOs and invite new comers to PL projects as much as possible!

We are planning for video calls I'm each month during SoC program. So, it will be great to see you all. It will be great if you can review each others proposals and give some suggestions.

Looking forward to it! I'll check on other's proposals too and see if I can help them in any way possible!

Also, it will be great if you want to (optional) to hold a conference at your university and explain Public Lab. What we do here? Programs and our workflow. This will be great help to public lab.

I'd definitely like to organize something like this ASAP at my uni as well! As a matter of fact, I always wanted to do something like this from the very moment I read this post and the amazing response it generated. We do organize meetups and such, and other than that I'm a technical assistant at the forementioned tech societies so organizing something like this shouldn't be cumbersome at all.

All participants all programs are encourage to write these in their proposals.

Write what out exactly? Sorry, but did I miss on including something in my proposal? I am unable to figure this out, can you please clarify?

Thanks again! Really appreciate it! πŸ‘

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


maybe I should stick to publishing my monthly progress directly to publiclab.org?

Blog post should be posted on Publiclab.org

what period (the current or the SoC one)

Blog work has to be done after you are selected in the GSoC at starting and end of each coding/working period. Does this makes sense?

did I miss on including something in my proposal?

Add some time for blogs, pr reviews and ftos in your timeline(about 3-4 days). Thanks

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Hi @rexagod, this looks really good...the way that you explained things is just awesome!!! πŸ‘ πŸ˜ƒ

Reply to this comment...


@icode365 Thanks! I'm glad you liked it!! πŸ˜ƒ

@bansal_sidharth2996 Thank you so much for the review! I've made the requested changes above and duly noted your suggestions! Can you please have a look? Much appreciated! πŸ‘

@warren I've tried to answer your query in my comment (below sashadev-sky's comment) above. Can you please check it out and provide your feedback on the same? Many thanks!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@rexagod amazing proposal. Completely love how you have tackled the whole thing in a great flow! Congratulations!

Reply to this comment...


@rexagod Thanks for sharing your interesting proposal. I have a question not related to the code but example pictures/gifs described above. It looks like your stitching the same picture? If this is this case, shouldn't it be a null result? Also, the road is aligned but portions of the road (below the intersection) are not matched? Should the non-match areas also determine when images are /are not stitched? Great coding BTW. Thank You

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Hey there @MaggPi!

That gif only shows the working of the AS module, and none of the images were actually wired to/utilized any ORB algorithm in that particular example. Let me explain.

The gif in question makes use of two images to demostrate only the automatic stitching ability of the AS module. In that particular example, I've explicitly passed the coordinates where the second image is supposed to be stitched to the first one. Of course we can do that using ORB, but for now I guess, the ORB isn't that advanced and may give some pretty messed up/unwanted results every once in a while, hence completely depending on it might not be the a good solution for now, so, as is visible in section (d), I'm thinking of displaying a modal that shows how good both the images compare, and leaving to the user if they want to stitch it in a left/right/top/down manner, and then automate AS to take up those coordinates and stitch the images in the desired orientation. Does that make sense?

Also, thank you so much for showing your interest in my proposal! Appreciate it! πŸ˜ƒ

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Login to comment.

Public Lab is open for anyone and will always be free. By signing up you'll join a diverse group of community researchers and tap into a lot of grassroots expertise.

Sign up