Support NeoGAF

[wolfmat]

How do I check if an element is already in a list/array?

I have the following code:

for item in page_info:
            if len(item.contents) > 3:
                a_element = item.contents[3].find("a")
                if a_element:
                    link = a_element.get('href')
                    
                if "/places" in link:
                    full_url.append(url + link)

I'm scraping a page for url links and placing them into "link." The url's coming from this part, look like "/places/blahblahblah" or "/events/blahblahblah." The last piece of two lines of the code are making sure it is only grabbing the "/places" urls.

The problem is, that there are duplicates urls on each page. How do I check if the url has already been scraped before adding it into the full_url array?

I would appreciate any help/direction.

So you're writing python there.
Sets guarantee that each item is unique.
If you add() an element that is already in the set, nothing happens.

url_set = set()
for item in page_info:
            if len(item.contents) > 3:
                a_element = item.contents[3].find("a")
                if a_element:
                    link = a_element.get('href')
                    
                if "/places" in link:
                    url_set.add(url + link)

Edit: You can also do what you did and then use the set to make the array unique once the for-loop is through:

full_url = list(set(full_url))

That's kinda wasteful though. Using a set in the first place is cheaper.

Edit: Note that mathematically, sets do not guarantee order preservation, but in python, they just _do_ preserve it. Just so you know.

Edit: Also note that you of course _could_ scan the array each time and append() conditionally:

if (url+link) not in full_url:
    full_url.append(url+link)

This sucks though because complexity grows exponentially with array size. You won't notice it with small-ish arrays (like, sub-10k elements) because "if x in list" will result in C list code for python being utilized. But things get shitty if you do this often, with big sites.
sets internally operate on hashes if I'm not mistaken (similar to dictionaries), so complexity of unique adding is linear (lexical lookup).

 

[GK86]

My only thought is to traverse through your full_url array with the new url value until you find a match or traverse the entire array and only append it if you don't find a match. It seems kinda inelegant and possibly time consuming dependent on how big your url list gets though.

That was my initial thought process, but I figured I was making it more complicated than it had to be.

So you're writing python there.
Sets guarantee that each item is unique.
If you add() an element that is already in the set, nothing happens.

url_set = set()
for item in page_info:
            if len(item.contents) > 3:
                a_element = item.contents[3].find("a")
                if a_element:
                    link = a_element.get('href')
                    
                if "/places" in link:
                    url_set.add(url + link)

Edit: You can also do what you did and then use the set to make the array unique once the for-loop is through:

full_url = list(set(full_url))

That's kinda wasteful though. Using a set in the first place is cheaper.

Edit: Note that mathematically, sets do not guarantee order preservation, but in python, they just _do_ preserve it. Just so you know.

Ahh OK. Thank you so much. Was about to ask about order preservation. I appreciate your help.

 

[wolfmat]

Sorry, I was mistaken. Order is not necessarily preserved. It depends on the type (I guess it boils down to the internal hash, so you might see alphabetical ordering with strings, for instance). (Edit: Nope, not even alphabetical.)
Therefore, you need a lookup set to check if you should append:

url_set = set()
for item in page_info:
            if len(item.contents) > 3:
                a_element = item.contents[3].find("a")
                if a_element:
                    link = a_element.get('href')
                    
                if "/places" in link and not (url + link) in url_set:
                    url_set.add(url + link)
                    full_url.append(url + link)

This preserves order and guarantees uniqueness. It's costly though.

Edit: You could also abuse OrderedDicts:

import collections
od = collections.OrderedDict()
for item in page_info:
            if len(item.contents) > 3:
                a_element = item.contents[3].find("a")
                if a_element:
                    link = a_element.get('href')
                    
                if "/places" in link:
                    od[url + link] = None
full_url = od.keys()

This is the cheapest way, I think. Use OrderedDicts if order is important.

 

[Two Words]

So, both my professor and the book we are using say that structures in C++ cannot have functions or initialize variables within the structure. But I tried making a structure that can have a constructor and methods for getting and setting data, and it works. I can initialize the variables with a default constructor or I can use an overloaded constructor. Is this a surprise to anybody else?

 

[poweld]

So, both my professor and the book we are using say that structures in C++ cannot have functions or initialize variables within the structure. But I tried making a structure that can have a constructor and methods for getting and setting data, and it works. I can initialize the variables with a default constructor or I can use an overloaded constructor. Is this a surprise to anybody else?

I think you need a new professor and book!

But seriously, yes, structs and classes are almost identical in C++. I think the difference is that often times structs are used for simple data collections and classes for more meaningful objects - mostly by convention.

 

[cpp_is_king]

So, both my professor and the book we are using say that structures in C++ cannot have functions or initialize variables within the structure. But I tried making a structure that can have a constructor and methods for getting and setting data, and it works. I can initialize the variables with a default constructor or I can use an overloaded constructor. Is this a surprise to anybody else?

Book is wrong. Are you sure the book says in C++? What you say is true in C, but completely wrong in C++.

I think you need a new professor and book!

But seriously, yes, structs and classes are almost identical in C++. I think the difference is that often times structs are used for simple data collections and classes for more meaningful objects - mostly by convention.

Interesting trivia. They are not identical. They mangle differently, so if you have a class Foo, and you forward declare it as struct Foo and then try to use it, you're going to have a bad time. I know 100% this is true on Windows using Microsoft's compiler and any other compiler which generates code with Microsoft's ABI. It may or may not be true for the Itanium ABI though.

 

[poweld]

Interesting trivia. They are not identical. They mangle differently, so if you have a class Foo, and you forward declare it as struct Foo and then try to use it, you're going to have a bad time. I know 100% this is true on Windows using Microsoft's compiler and any other compiler which generates code with Microsoft's ABI. It may or may not be true for the Itanium ABI though.

Well... I did say almost identical. I've never tried interchanging the two, but I suppose it's good to know that it's not, in fact, allowed.

 

[cpp_is_king]

Well... I did say almost identical. I've never tried interchanging the two, but I suppose it's good to know that it's not, in fact, allowed.

The other difference, btw, is that the default access level of a struct is public, and of a class is private

 

[Two Words]

Book is wrong. Are you sure the book says in C++? What you say is true in C, but completely wrong in C++.



Interesting trivia. They are not identical. They mangle differently, so if you have a class Foo, and you forward declare it as struct Foo and then try to use it, you're going to have a bad time. I know 100% this is true on Windows using Microsoft's compiler and any other compiler which generates code with Microsoft's ABI. It may or may not be true for the Itanium ABI though.

Isn't that because structs default to public members and classes default to private members?


Edit- looks like you just said that.

 

[cpp_is_king]

Isn't that because structs default to public members and classes default to private members?


Edit- looks like you just said that.

It's not because of that, no. Whether something is public or private has no effect on generated code. The fact that structs and classes mangle differently was just an arbitrary choice they made when designing their ABI.

I mean yes structs are public by default and classes private, that's just unrelated to the fact that they mangle differently

 

[Telephoto999]

So GAF.. I am a relatively new programmer and have gotten comfortable with enough languages and various editors / IDE's to take on some projects of my own.

Where I am lost... are what tools to use before I actually start to code. I am referring to UML tools to design my classes... etc. Can you guys recommend a tool that will assist with the pre planning of application design? I will be writing small applications for friends and family to start, but I want to do them the correct way. I'd like to plan them out, document them visually before just jumping in and coding my way out.

Thanks!

 

[Moosichu]

So GAF.. I am a relatively new programmer and have gotten comfortable with enough languages and various editors / IDE's to take on some projects of my own.

Where I am lost... are what tools to use before I actually start to code. I am referring to UML tools to design my classes... etc. Can you guys recommend a tool that will assist with the pre planning of application design? I will be writing small applications for friends and family to start, but I want to do them the correct way. I'd like to plan them out, document them visually before just jumping in and coding my way out.

Thanks!

It might sound quite basic. But pen and paper is what I always use.

 

[Telephoto999]

It might sound quite basic. But pen and paper is what I always use.

I've actually seen this suggested on other forums. As a result, I looked around my house today and couldn't even find a notebook. smh.

 

[Treechopper]

I may have asked this before, but does anybody have any experience bootstrapping their own program on a system with minimal OS presence? I know you're supposed to set up the CPU stack and other registers, but I'd like to see some examples.

 

[Schweinehund]

Funny coincidence. I'm in a similar situation. We have to do two assignments: For the first, we're free to choose any object-oriented language, but the second has to be in Smalltalk using Squeak. And both assignments are actually games too.

I think I'm in a solid position now (took me forever to figure out how to quit my program from code).

This helped me a lot with squeak.

 

[cpp_is_king]

So GAF.. I am a relatively new programmer and have gotten comfortable with enough languages and various editors / IDE's to take on some projects of my own.

Where I am lost... are what tools to use before I actually start to code. I am referring to UML tools to design my classes... etc. Can you guys recommend a tool that will assist with the pre planning of application design? I will be writing small applications for friends and family to start, but I want to do them the correct way. I'd like to plan them out, document them visually before just jumping in and coding my way out.

Thanks!

Been in the industry for close to 20 years. Never seen a single person actually use UML

 

[Telephoto999]

Been in the industry for close to 20 years. Never seen a single person actually use UML

Fair enough, that is actually a relief honestly as I don't like how it's formatted. I just figured that was the "right" way. What is the best practice for mapping out your applications before you start coding?

 

[wolfmat]

Fair enough, that is actually a relief honestly as I don't like how it's formatted. I just figured that was the "right" way. What is the best practice for mapping out your applications before you start coding?

You need to know your data, how you want to modify it and how you represent it.
How you write these basics down is not really relevant. What matters is that you
- Make a statement what the application is supposed to solve
- Design the data structures that are necessary to solve this
- Sketch out how to access the data to read and write
- Roughly scribble interfaces

Once you've done that, you should investigate what parts of all that you can directly achieve with libraries. The libraries will most likely force you to use certain data formats, interfaces and such. Modify your design to integrate the libraries.

Now, you're already well-equipped to cough up a prototype. Hack the prototype to learn fast what you did wrong in your design. Write these lessons down and modify the design.

Once you're confident you've understood everything your prototype could teach you, start from scratch with a "proper" implementation.

 

[Granadier]

You mean the proper way isn't to just start mashing code in with a rough guesstimate to the design sitting in your brain?

 

[wolfmat]

You mean the proper way isn't to just start mashing code in with a rough guesstimate to the design sitting in your brain?

That's also nice. It's fun to just start mashing code. It's less boring than doing the planning bullshit, certainly.
Sometimes it's way more awesome to just start to type.

It gets way better with more experience with properly planned projects though.

 

[Slavik81]

You mean the proper way isn't to just start mashing code in with a rough guesstimate to the design sitting in your brain?

Personally, I took this approach for years. I tried to map things out, but I had literally no idea how big programs were structured. Actually building the program (iteratively) was the only way for me to figure out what was needed.

Been in the industry for close to 20 years. Never seen a single person actually use UML

I've never seen actual UML, but I certainly saw and drew a lot of boxes with arrows.

 

[GamerSoul]

Here's a summary:

Lol that's amazingly accurate from what ive heard in some cases.

 

[Granadier]

Personally, I took this approach for years. I tried to map things out, but I had literally no idea how big programs were structured. Actually building the program (iteratively) was the only way for me to figure out what was needed.

This is my usual process too when I'm working on something right now. At this point there are so many topics and projects out there that I have no clue about. If I'm tasked to work with one I find it more effective to start hacking something together while Googling/reading the documentation, than to sit down and try to plan the whole thing.

If I'm refactoring or working on something I've done before though I will usually sit down and make at least a simple overview of what I predict will be needed for the project.

 

[Hypertrooper]

I guess it is the wrong thread, but for me personally it is the closest one to post this.

At the end of a lecture I am doing a short summary of the content with LaTex. Right now, I am doing one for Software Engineering and heading to a problem to create UML diagrams. I have no clue which package to use because I am pretty new to LaTex. Does anyone have a recommendation?

 

[oxrock]

For a fun project I decided to make a strange take on an evolution simulation. You start out with x number of generic/identical boxers and they fight in tournaments over and over again while repeatedly having the losers weeded out and replaced by boxers with an averaged stats of the parents but with minor chances of minor mutations. The program appears to be functioning but the results are so strange that I had to come here and ask for other people to take a look.

The program is here: http://pastebin.com/AjmmfLv6
The code is commented to give a pretty decent idea of what is going on, I'm not used to sharing my code so it may not be perfect. So if there's any questions, feel free to ask.

I'd love if some coding guru's would take a look at both the code and the information gained from running some tests and let me know what they think is going on.

Important: A small sample size (less than 100 boxers) isn't suggested. Whether a boxer is male or female is completely random and without a male and female parent there can be no boxer babies. I also find running 1000 tournaments to give a better polarized view of the stats trend than simply 100 iterations for example.

The code is written in python 2, the only major changes i'm aware of to make it work properly in python 3 is modifying the print statements.

 

[Wolf Akela]

Started to read lightly about design patterns just to brush up for interviews/tests, since I thought I had to fill this big blank hole in my skillset. I ended up realizing I was already implementing several of them without knowing, and that it's nearly impossible to avoid them.

I mean, facade itself is basically APIs. It's nearly impossible to avoid decorators in JavaScript unless you consciously avoid using it. I felt dirty when I started writing factories but then I learn what it's called and realize they're not that uncommon.

 

[Skittles]

Gonna kill myself while trying to program this CPU scheduler

 

[JesseZao]

I know C/C++ isn't really vogue in enterprise development, but I found a cool series to learn it while some guy makes a game from scratch. He has 90+ hrs of 1hr videos to catchup to his currrent progress, but I think I found a new fun resource for my downtime at work.

https://handmadehero.org/

He streams an hour on twitch and it get uploaded onto youtube later. Looks like a fun way to be exposed to C and some C++ (and get an idea of how to approach project design).

 

[Aureon]

For a fun project I decided to make a strange take on an evolution simulation. You start out with x number of generic/identical boxers and they fight in tournaments over and over again while repeatedly having the losers weeded out and replaced by boxers with an averaged stats of the parents but with minor chances of minor mutations. The program appears to be functioning but the results are so strange that I had to come here and ask for other people to take a look.

The program is here: http://pastebin.com/AjmmfLv6
The code is commented to give a pretty decent idea of what is going on, I'm not used to sharing my code so it may not be perfect. So if there's any questions, feel free to ask.

I'd love if some coding guru's would take a look at both the code and the information gained from running some tests and let me know what they think is going on.

Important: A small sample size (less than 100 boxers) isn't suggested. Whether a boxer is male or female is completely random and without a male and female parent there can be no boxer babies. I also find running 1000 tournaments to give a better polarized view of the stats trend than simply 100 iterations for example.

The code is written in python 2, the only major changes i'm aware of to make it work properly in python 3 is modifying the print statements.

I may not be 100% correct, but have at it:
I think it's rounding. Python rounds down.
I'm not going to go through your fightnng code to see if it's correct (Unit test it!), but that's my best guess.

All your stats are like this:
self.base_endurance = (parent1.base_endurance + parent2.base_endurance)/2
This means that, for example, if you're getting 8+13, you're back at 10 - and if you're getting 9+10, you're down to 9.

Also, batman, array those stats!


Also, i couldn't help myself (even if really, really, really should be working on other code), and did the simple fix myself:
http://pastebin.com/C2E7yKC5

So, yeah, guess was correct.

Setting the range to -2 to +1 nets a even all-10 winner: The explaining of this is left as an exercise to the reader.

Now, repeat with me: I WILL NOT DIVIDE INTEGERS LIGHTLY.


PS: I love how even beginner-written Python is very readable.
PPS: Please don't do this. Providing code is good, but don't ask someone else to actually run it to see what's wrong.

 

[oxrock]

I may not be 100% correct, but have at it:
I think it's rounding. Python rounds down.
I'm not going to go through your fightnng code to see if it's correct (Unit test it!), but that's my best guess.

All your stats are like this:
self.base_endurance = (parent1.base_endurance + parent2.base_endurance)/2
This means that, for example, if you're getting 8+13, you're back at 10 - and if you're getting 9+10, you're down to 9.

Also, batman, array those stats!


Also, i couldn't help myself (even if really, really, really should be working on other code), and did the simple fix myself:
http://pastebin.com/C2E7yKC5

So, yeah, guess was correct.

Setting the range to -2 to +1 nets a even all-10 winner: The explaining of this is left as an exercise to the reader.

Now, repeat with me: I WILL NOT DIVIDE INTEGERS LIGHTLY.


PS: I love how even beginner-written Python is very readable.
PPS: Please don't do this. Providing code is good, but don't ask someone else to actually run it to see what's wrong.

*sigh* my noobness is so easily revealed. Thanks for taking the time to debug and test this, I had pretty much given up trying. I'd tested everything I could think of and nothing corrected the problem.

 

[Aureon]

*sigh* my noobness is so easily revealed. Thanks for taking the time to debug and test this, I had pretty much given up trying. I'd tested everything I could think of and nothing corrected the problem.

May i direct you to Our God And Savior, Unit Testing?
Ideally, in those cases, what you want to do is test out every potentially problematic function with dummy stats. That's unit testing.
You'd have found out, somewhere along the line, that the issue was in the create children function, and that it swung low.

Homework: Think of how to plan a test that would catch that : )

 

[Slavik81]

When using vim, if I do :make I have a hard time finding where my output begins when I have a lot of errors. Any idea how people usually handle this?

It seems the command I was hunting for is :copen. Or, :cw for both open and close.

Started to read lightly about design patterns just to brush up for interviews/tests, since I thought I had to fill this big blank hole in my skillset. I ended up realizing I was already implementing several of them without knowing, and that it's nearly impossible to avoid them.

I mean, facade itself is basically APIs. It's nearly impossible to avoid decorators in JavaScript unless you consciously avoid using it. I felt dirty when I started writing factories but then I learn what it's called and realize they're not that uncommon.

Yeah, the valuable part of design patterns is really just having names to describe the various patterns you see in your code. They help programmers communicate.

 

[Terrordactyl]

So I'm pretty new to programming in general, for a course I have to create a calculator, which I've already done, but now I have to enhance it. I have to create a class for addition and subtraction, then inherit and extend it into a class for multiply and divide, and I don't really get how that's really practical. It might be the way I have my code set up already for the operations, but if anyone could add anything to get me to wrap my head around this better that'd be awesome.

 

[close to the edge]

So I'm pretty new to programming in general, for a course I have to create a calculator, which I've already done, but now I have to enhance it. I have to create a class for addition and subtraction, then inherit and extend it into a class for multiply and divide, and I don't really get how that's really practical. It might be the way I have my code set up already for the operations, but if anyone could add anything to get me to wrap my head around this better that'd be awesome.

That does sound slightly odd. If you are required to implement the basic arithmetic operations yourself that would sort of make sense, since multiplication is iterated addition and division is basically iterated subtraction. But then inheritance still wouldn't really make sense.

 

[injurai]

So I'm pretty new to programming in general, for a course I have to create a calculator, which I've already done, but now I have to enhance it. I have to create a class for addition and subtraction, then inherit and extend it into a class for multiply and divide, and I don't really get how that's really practical. It might be the way I have my code set up already for the operations, but if anyone could add anything to get me to wrap my head around this better that'd be awesome.

Perhaps the point is to get you thinking about inheritance, not so much as saying this is the best way to solve the problem. The problem is a vehicle for practicing a particular design pattern.

 

[MiszMasz]

Perhaps the point is to get you thinking about inheritance, not so much as saying this is the best way to solve the problem. The problem is a vehicle for practicing a particular design pattern.

That's what it looks like to me as well.

Don't worry too much about the wider logic or any real-world application, focus on the spec for the class hierarchy and implement methods as required.

Without knowing more, it seems something along the lines of one class with methods for addition and subtraction, then a subclass that inherits those methods and adds a couple of its own for multiplication and division. Then make an instance of the subclass and test all methods - inherited and new.

 

[Saprol]

Spoke with a recruiting agency recently and they include an automated technical assessment as part of their process. Seems odd given how a lot of the questions focus on whether you remember Java syntax/methods of a particular class. I can't imagine people with professional experience caring about memorizing this sort of stuff instead of just looking it up if they need it.

At the Advanced level, the candidate will be capable of working on projects involving Java 7 and will be capable of mentoring others on most projects in this area.

Yeah, I suppose I can mentor people on projects involving guessing on multiple-choice questions. Not so much on all that stuff they asked me about though.

 

[Randolph Freelander]

^ That looks like a Brainbench result. Yeah, it's worthless.

 

[injurai]

I know C/C++ isn't really vogue in enterprise development, but I found a cool series to learn it while some guy makes a game from scratch. He has 90+ hrs of 1hr videos to catchup to his currrent progress, but I think I found a new fun resource for my downtime at work.

https://handmadehero.org/

He streams an hour on twitch and it get uploaded onto youtube later. Looks like a fun way to be exposed to C and some C++ (and get an idea of how to approach project design).

Thanks for reminding me about this. I keep meaning to go through it. Was actually thinking I might try to learn rust then go through it to implement similar things.

Just need to find the time. Time... never enough of that.

 

[Slavik81]

I'm considering renaming my github and stackoverflow accounts. Right now they're both slavik81, but my initials are available. I'll probably be looking for a job in a year or so. Any thoughts?

 

[Randolph Freelander]

I'm considering renaming my github and stackoverflow accounts. Right now they're both slavik81, but my initials are available. I'll probably be looking for a job in a year or so. Any thoughts?

Most people on SO just use their names, do they not? I do. The alias thing seems best reserved for forums and such, but SO is much like social media, at least in the naming department. It's part of your brand.

 

[JonathanNemo]

I'm considering renaming my github and stackoverflow accounts. Right now they're both slavik81, but my initials are available. I'll probably be looking for a job in a year or so. Any thoughts?

Up to you.

The trouble with operating under a real name with a full time job is that you may run into issues where you're expected to act as though you're speaking for the company, in a public setting. Any software written while employed might also fall under your employer's domain, and that's a sticky thing to get out of unless you're willing to clear things with HR.

Contractors have some restrictions too, but there's much more leeway for projects and work on the side. Also general life PITA things to worry about such as taxes and insurance that come up as an independent contractor.

I think SO's alright, as long as you realize that working for a larger entity might restrict your future activity there. Source code repos, that's basically your resume so some stuff under your real name makes perfect sense.

Don't feel you have to go all in one way or another, internet pseudonym or real name.

 

[Yoda]

I have a set of interviews with Twitter in a few days for a Software Eng internship. Has anyone here interviewed with them before? And if so any information on the particular flavor of CS/DS questions they ask would be much appreciated .

 

[WanderingWind]

Nobody in my life understands how great it feels to finally find a solution to a problem. I finally got a lightbox element to work correctly and it feels like victory to me.

 

[lorebringer]

Nobody in my life understands how great it feels to finally find a solution to a problem. I finally got a lightbox element to work correctly and it feels like victory to me.

 

[Angelus Errare]

Nobody in my life understands how great it feels to finally find a solution to a problem. I finally got a lightbox element to work correctly and it feels like victory to me.

They'll never know...a shame too.

 

[Aureon]

Nobody in my life understands how great it feels to finally find a solution to a problem. I finally got a lightbox element to work correctly and it feels like victory to me.

Eureka Moments are why we program, truly.

 

[Treechopper]

Eureka Moments are why we program, truly.

Well, that and all the chicks we impress with our talk of systems integration.

 

[Rur0ni]

Well, that and all the chicks we impress with our talk of systems integration.

And 100% test coverage.

 

[injurai]

rust now in beta!