There's being taught and then there's exploring and practice. It's the exploring and practice that is often missing today, IMO because we all want everything to happen so quickly.

In my sport of choice, jiujitsu, there's been a shift happening in teaching. Traditionally the teacher would show a move or two then have the class drill the move. The shift now is more conceptual where the teacher might say the goal is get chest to chest and then have the students operate at 20% figuring it out on their own. While there are some pretty vocal camps on both sides, I think having both methods is the best to learn from. Being taught a solution to a problem a student discovered on their own during discovery is when learning really takes hold IMO.

Finally, the best way to really learn that I have found is to teach something.

No it's not; this is the point of apprenticeship. It's just not compatible with the educational institutions people actually pay for.

> I think the learner would just have to go lick a lot of things for themself.

This is not contradictory. If your idea of teaching is just lecturing at people and occasionally verifying specific knowledge, obviously they're not going to learn very much. Your job as a teacher is to facilitate a growth curve of increasingly more difficult and unwieldy problems to solve.

You are an expert. But if you wanted to, how would you teach that? I think the learner would just have to go lick a lot of things for themself.

This is such a weird and amazing analogy. Thank you for sharing it! I don't know if I've ever heard this concept explained in quite this way before.

My intuition is that creativity is a more of a feeling one has, rather than the result of some teaching method or cultural background. It seems very closely related to one's interests and also the novelty effect. Just think about the first time you experienced something, like a great song or your first love. The mind is just bubbling with thoughts. How can that feeling be experienced again with the same object like it was the first time? It can't. It's never the same as the first time. Even if we can mentally forget, the subconscious somehow remembers it's had that experience, so it isn't fresh anymore.

So, yes, you need to make connections between existing things to come up with something new, but once the fascination with a thing dissipates and it becomes familiar, any further creativity in that area quickly dies off. Our interest just moves on.

Perhaps the really creative people are just the ones who have this ability to see familiar things with ever fresh eyes?

You don't want a pilot who is creative when it comes to max takeoff weight.

Obviously there are good reasons to memorize certain things, creativity just isn't one of them.

But there are plenty of people who can pass tests and are terrible engineers. When people talk about memorization they talk about remembering the words without understanding the concepts, such a person is no better off than the person carrying around a text note with the concept written down.

When you understand these concepts you will remember those formulas, but memorizing the formula doesn't make you understand it. Therefore creativity doesn't fundamentally comes from memorization, there is something else there.

This is just confirmation bias. Either you're in a field where you must have a degree, so everyone who couldn't pass tests simply never became an engineer (eg EE) and whether or not the test is a good measure is irrelevant because it nonetheless gates your sample

OR you're in software engineering and the people who struggled on tests work beside you, but they don't tell you about their past performance on tests because you have a chip on your shoulder and they don't want you to look down on them.

I failed a lot of tests in college and now I'm a great software engineer.

Most of the time memorization is a key role for creativity, the easier you can jump between ideas the more combinations you can explore (seems like the brain is constrained by some cache bottleneck in a way).

This is not mere memorization. The GP's point that not everything that makes you remember this is simply "memorization", and certainly all of those things contribute differently to creativity.

> I also have some written tests for certifying pilots.

Yes, piloting is one of the tasks where you absolutely need lots of random information memorized. You should also not do a lot of "creativity" in it.

The mnemonic to remember V = I * R is as a triangle with V at the top and I, R on the second row, the same way you remember the relation Distance = Speed * Time or Mass = Density * Volume in high school. Or any other triangular multiplicative relation.

> If you can't remember the pieces making up a concept, how are you going to remember the concept?

You touch upon the different levels of knowing it. Yes, having to carry a card with the formulas on it shows no knowledge. But, if you have to memorize the formulas, your knowledge is still not adequate. You're just regurgitating a formula that you memorized so you can plug in numbers. You don't understand the "why" of Ohm's law. Of course voltage is equal to current times resistance, it's obvious by what these things are! It should be as self-evident as "Of course distance is equal to speed times time!"

Another example: You can have the Lorentz transformation formulas memorized but still not really understand the "why" of Special Relativity.

...I mean, I didn't even go to that great of a college and no one would have made it past the second year of EE without memorizing 10x more formulas than that.

I'm just completely lost on how it's even possible to have an EE degree and needing a card. Signal processing classes required math 100x more difficult than that. I had to know quaternions by my second dsp class.

To most people I’d wager “memorize” has a strong connotation for the synthetic version only, with an emphasis on a stripping-out of context.

I recognize that stripping away context can be valuable—-drilling a tennis serve over and over outside the real-time context of a game is extremely helpful.

Flashcards are rarely valuable in the same way. For semantically oriented tasks, an impoverished context is usually not very helpful. Receptive skills like letter and character recognition might be an exception. But even then you’ve got to make the leap to reading at some point.

> no one adds or multiplies enough in daily life for natural memory formation

To the contrary, there’s a fascinating study of children in South America who had very fluent mental math skills for making change because they sold fruit on the side of the road. They couldn’t solve the exact same problem in story problem format in a classroom, though. Synthetic contexts usually don’t transfer well to real life.

If anyone wants some primary school appropriate word problems, let me recommend the collections by Lenchner, e.g. https://archive.org/details/mathematicalolym0000lenc

Also try Kordemsky’s Moscow Puzzles https://archive.org/details/boris-a.-kordemsky-the-moscow-pu...

I think in common usage the word "memorize" very strongly implies that it's the lossless storage and retrieval of some highly specified sequence of information. No amount of studying American Civil War history would be referred to as "memorizing the Gettysburg Address" unless you could recite the speech word for word.

Isn't that kind of the point for the other side?

Nobody memorizes the answer to 212+457. There are no flash cards for every possible addition up to infinity.

In my experience, the more people think that math is something to memorize, the worse they are at math.

To memorize means to commit to memory. It is an action. But memories are created by not just actions, but experiences. Experiences can create memories without you having committed them; without you having memorized anything.

This isn't actually a settled matter. I did a literature dive a while back and found that drilling vocabulary flashcards shows the highest benefit on artificial recall tasks (like multiple choice tests), and over relatively short time scales (days to weeks). Studies that looked at longer time scales (months to years) and more organic tasks generally showed mixed results. Which I generally interpret as a sign that the literature in question is highly susceptible to the file drawer effect.

And that in turn suggests that the magnitude of flashcarding's value for this kind of thing has a lot to do with your goals. In a nutshell, are you more interested on the science on how to develop communicative proficiency over the long run, or are you more interested on the science on how to get a good grade in class?

I'm studying Chinese right now, and I do use flashcards, but it's not because I believe it's the best way, per se. It's because it's a convenient option for reviewing characters and words that don't appear often in the reading material I have. When they do appear often in my reading material, I find (anecdotally) that it takes a lot fewer organic repetitions to get the character or word to stick in my head than it does with flashcard repetitions.

It's also worth mentioning there's no particular reason to assume Chinese and Japanese schools are any less likely than schools elsewhere in the world to cling to inefficient pedagogical techniques out of a sense of tradition. So one can't necessarily assume that the way they are doing it is the way they ought to be doing it.

Also it takes a surprisingly long time for children to learn language. I used to think it sorta-kinda happened over a year or two, but having children myself revealed how wrong I was.

I have written up a fairly hefty dictionary of words they mispronounce or even just invent on their own because they don't know the one in their native tongue. My oldest rarely contributes individual words to this dictionary anymore, but he still improvises expressions and idioms.

Right—not simply memorizing what's happening around them. Those are fundamentally different activies. That is the gist of the parent comment's point.

They've also acknowledged expressly that rote memorization techniques are "valuable" and "have a place."

Only if your goals don’t include being able to speak or write.

> You can’t be serious advocating “just go read stuff”

True, not “just.” And Chinese is particularly tricky because the ideograms convey little to no phonetic information. Even so, almost any activity I can imagine seems superior to traditional flashcards. Photo flashcards (vs. translation), listening along to highlighted text or closed captions, deciphering street signs or memes, even the written drills you mentioned. (Or better, “write 5 phrases that all start with character X”). Our brains crave meaning, and flashcards offer very little of it.

> even native speakers learn them by repetition

No argument there, most everything is learned by repetition, but I’m interested in context. Native speakers already know the verbal form of most words they’re learning to write, even in Chinese. I’d argue the meaning is stronger.

> life of a toddler is vastly different than an adult

True. The scale of their learning tasks are much bigger than ours. They have to learn that they exist, that their family exists, that they can vocalize, that language is a thing, that they want and need things, and that they can get them by communicating.

> An adult has much less time for that

I think this is a good entry point to the core of the issue for me—-small children don’t “set aside time to learn,” they just learn. You and I do this also, though it’s less novel and flexible and therefore maybe less salient. I think we place too much value on structured or synthetic learning as “real” learning when in fact it’s often extremely inefficient compared to our natural learning tendencies.

There’s a spectrum of structure, starting with what we choose to pay attention to, to an open-ended “study time,” to guided classroom activities, to timed math drills. Flashcards are at the extreme reductivist end of that spectrum. I suspect we like them because they’re easy to understand, uniform, predictable, and convenient to create and use. Creating more effective learning opportunities and supports is substantially harder, but generally worthwhile IMO.

And plus, general recommendation is to learn words elsewhere and just put them into anki to not forget.

Some people like it, but it is not the only or the most recommended way to learn speak and write.

I've tried this for years with Japanese kanji and never really got very far. Just didn't work well, they largely were just a big blob of lines.

Then I found an Android app (Kanji Study) that mixes this in with informational screens that break down kanji into radicals and puts them alongside a bunch of multi-kanji words and uses them in sentences so we can see them in context, and it's actually been working.

I've a toddler who can read 3 paragraphs of 3 sentences each, and then tell you the details of the story he read[1]. He is 4y6m, right now. He has never learned the alphabet or the names of the letters (A,B, C, D, etc). He has only learned the sounds a letter or sequence of letters make for certain patterns.

You most definitely do not need to memorise the alphabet in order to learn to read!

Teaching children the alphabet before teaching them reading makes it a lot harder for them to learn actual reading.

[1] I've seen kids as old as 7 get confused by a book with no pictures, and he sails right on through because I taught him to read (using the DISTAR alphabet), and made sure none of our daily lessons had even a single picture in it.

I’m all for hard work; learning is usually very hard work. I also think we need expert guidance.

But let’s make the difficulty useful/effective rather than counterproductive.

Also, to answer your questions in a very simple way: the entire reason you even became an expert is because another expert somewhere along the way gave you an easy in to the knowledge, they coddled you. This is why you can "stare at a compiler error for a few hours before figuring out what's broken". Without that expert, you wouldn't even understand what a compiler even is.

What I find instead, is that by just processing novel information, especially if I focus on analysing it, my brain internalizes insights and builds model of that type of thing, allowing me to either imperfectly reconstruct what I've seen, or to come up with an infinite array of permutations, extrapolations, etc which is where the real ideas come from.

Further, ideas crucially revolve not around just the information itself, but the "feel" for what role they play in the whole, how well they do it, in what way they're notable, etc.

In fact I'd straight up claim that memorization is antithetical to creativity - a perfect ML autoencoder or GAN would just regurgitate the training data. Creativity comes from generalisation while memorisation is analogous to overfitting.

Expertise is the result of learning from past experience, both in developing an internal intuition for what you're doing and in having past patterns to draw upon. To the extent that experts have simple easily verbalizable heuristics, these are largely post-hoc attempts at explaining their intuition rather than an accurate reflection of how they make decisions.

And, in fact, experts can't even always do that: it is perfectly possible for experts to make good decisions without being consciously aware of why they are making them, and explaining how to make good decisions is a separate skill from being able to make them in the first place. The book I mentioned has a memorable story about a firefighter who thought he had precognition after pulling his team out of a dangerous situation without any specific indicator of the danger, but I figure a more common example is experts saying they did something because it was the "obvious" or "clean" or "better" way to do it and getting a bit flustered when pushed further.

We can see this in action pretty clearly if we look at advice for, say, writing. There is a lot of advice from good writers but just memorizing and blindly following this advice is actively counterproductive. Advice you can memorize fundamentally must lack nuance and context. We can see this clearly because so many different pieces of writing advice contradict each other and because good writers do not follow any of those suggestions with any consistency.

The same definitely applies to programming, which is why we have both "don't repeat yourself" and "you ain't going to need it", and why new programmers trying to apply either rule (or both!) to a codebase inevitably create a mess. What I've found with programming advice is that most suggestions are either actively wrong or too vague to be useful. (By the time you've learned enough about programming to be able to follow the vague advice, you don't need it very much!)

The only reason I can see is if you think memorization could be a shortcut to good taste, which it can't. Acquiring good taste requires broad experience—more information than you can possibly remember—such that you retain a suite of sophisticated intuitions. Cutting that information down to something that can be memorized would require you to (1) already have the intuitions you're seeking to acquire, and (2) be able to express them all in plain English, which, as far as I know, cannot be done. No painter has ever expressed their aesthetic in such a way that a student could memorize that expression and then have the same creative sensibilities as the original painter.

Ultimately, there's no substitute for the process of simply consuming lots of art while paying close attention to what you like about it.

seems to me there is a relatively big inductive gap there, you believe that there is an internalized heuristic and at its core is memorization, you may even have some evidence that this internalized heuristic has strongly informed your development of taste, but it is pretty difficult to make an argument that is the case for all people.

Aside from that I would say that "internalized heuristic with memorization as the core" puts everything on nurture and no input of nature - which I am pretty much in the camp of combinations of nature and nurture creating the person - of which taste must surely be a big component.

Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes

https://arxiv.org/abs/0812.4360

In the case of creativity it may be easier to make an argument - but surely you can find people who seem more creative with less ability to memorize.

Although I'd say traditional mnemonic devices like memory palaces are basically linear information storage and recall devices. This can create issues in building a flexible web of information, because loci or the order of the path can become dependencies and you can run out of unique spots in a given space, leading to memory interference.

Even spaced repetition methods (e.g. Anki) tend towards fragmentation of micro-ideas. Its perfect for terms, languages, and simple one question -> one answer ideas.

I've found a hybrid method of images, nested loci and spaced repetition to be most useful, because its flexible over time, and preserves relationships of ideas.

(Context: I co-founded a SaaS in this space: www.sticky.study)

Have never heard of this book but adding to my list now!

Borges was incredibly talented, but it's worth keeping in mind his dad was a writer too.

Good art very much relies on being exposed to lots of other good art first. I don't know that rote memorization is the best way to achieve that, but you definitely need that exposure.

For example one thing that puzzles me in western education is this reading comprehension. What exactly is this for? I studied in Russian school/university and we had nothing like that.

Reading means you should understand what you are reading, reading without comprehension is just nonsense for me.

Math anywhere in the world is all about how many patterns you know/seen. And inventing new math is all about changing one small thing about the stuff you already know and see if it remains logically consistent.

Knowing what to change largely depends on the stuff you already know. To that end even if you don't memorize facts, you still need to memorize patterns.

You have to memorize something. Nobody invents the Apple pie from scratch.

It's not that I set out to understand this specific thing but that I had studied statistics with flashcard support for a year and that happened to work after a few attempts.

My thought is: You're always practicing a behavior whether you like it or not.

Your mind is always setting you up to do more of whatever it is you're doing now. Both physical and mental. To do it more efficiently. With more ease. With more frequency. etc.

It's good motivation for me to mitigate a lot of negative behaviors. Angry in traffic, self anger, etc. If i can reason at least, of course. As i'm not interested in doing a lot of things more than i am, so i should avoid doing them now - if possible.

Meorising the derivation makes it easier to derive a second-order derivation, and so on. At some level of abstraction, going from first principles becomes prohibitively expensive and caching intermediary results, or so to speak, unlocks that again.

The trick is not so much which software or settings one uses, but writing high-quality prompts.

How so? Can you share what you have read and that is relevant and applicable in real life?

> There's a bad trend, Hacker News gets this kind of blog-style self-promotion every month that gets much comment attention, but the essays are not well-researched and with made-up assertions written by programmers talking out of their lane ("Engineer's disease") and not having done homework on the subject.

You are on the wrong site, this is not a scientific journal there is no need for scientific rigor in every post and comment.

P.S. The real life world is full of events and things happening if you cannot learn by yourself (create theories models on how the world works aka. pseudoscience apparently) and need academic verification for everything. Then you some kind of disorder :/ good luck

[1] Do you think that ChatGPT can reason? https://www.youtube.com/watch?v=y1WnHpedi2A

I disagree that you have to be exposed to a lot of music to even have a chance at coming up with something. Maybe coming up with something good, sure, but not having that spark of inspiration. People can have flash inspiration for garbage ideas too, it happens all the time in startups :P

That being said, the spark might happen more often the more you practice doing something about it, to your point.

Reiner Knizia is the most prolific board game designer of all time. He has 742 games credited to him on BoardGameGeek (some of those are rethemes of older games of his, but he still has probably at least 500 different games released).

About half of my prototypes are about on the level of cleverness and completeness as the lower rated half of the games he has published, at least, and maybe even a couple might slip into the top half somewhere (at his most brilliant his games are way, way better than mine though). But his name carries a lot of weight so he doesn't struggle to find publishers for a good chunk of the games he comes up with.

Meanwhile I've mostly switched back to video game development because at least that way I can easily publish the game myself when it's ready (game development is super hard in other ways, though). I tried for six years and got one signed board game that still hasn't been released after four years. Meanwhile there was a time where I was releasing 1-2 Flash games a year. If I had just made video game versions of my game prototypes instead I probably could have some basic playable version of it out on a website somewhere.

Although I do still submit some board game designs to unpublished game design competitions (I was a finalist in one of the biggest ones, for a design which still hasn't found a publisher) and go to conventions and set up meetings with publishers when I can afford to do so (I couldn't afford to this year, my spouse took a break from work and our income cut almost in half).

I love his pragmatic and rational approach.

Training an LLM on the entirety of knowledge at this dawn of humanity and, even if you give it literally infinite training time, it's never going to go anywhere. It's going to just continue making relatively simple recombinations of its training set until somebody gives it a new training set to remix. This remix-only nature is no different with modern knowledge, but simply extremely obfuscated because there's such a massive base of information, and nobody is aware of anything more than a minuscule fraction of it all.

---

As for the 'secret' of LLMs, I think it's largely that most language is extremely redundant. One thought or point naturally flows.... why do I complete the rest of this statement? You already know exactly what I'm going to say, right? And from that statement the rest of my argument will also mostly write itself. Yet we do write out the rest, which is kind of weird if you think about it. Anyhow the point is that by looking at language 'flow correlations' over huge samples, LLMs can reconstruct and remix arbitrarily long dialogue from even the shortest of initial inputs. And it usually sounds at least reasonable, except when it doesn't and we call it a hallucination, but it's quite a misnomer because the entire process is a hallucination.

I could see that -- agreed that "creativity" can be too blunt of a word to use for all situations

I don't think that meets the commonly accepted definition of memorization.

E.g. https://dictionary.cambridge.org/dictionary/english/memoriza... Memorization: the act or process of learning something so that you will remember it exactly.

If you are transforming the knowledge, i dont think that is the same as memorizing it. In fact, i think most people would describe that as the opposite of memorization.

Using your definition, everything would be memorization. For example, you could describe picasso as simply pattern matching on other paintings to make something new, but i think it would be crazy to describe his work as an act of memorization (then again, the way AI generated art works... maybe its not so crazy)

That question has a false precondition baked in. If you know how, it's not creative.

> It's some form of pattern matching right -- which imo is just a less obvious form of memorization

No. Sensing and matching patterns does not imply memorization. Everything you're saying is completely loaded.

Did you just discover memorization? Because the pattern I see in your words is similar to anyone who's just learned a new tool or technique - they overapply it everywhere as they learn to use it.

That is, creativity fundamentally comes from internalized knowledge (as the article says) but internalized knowledge doesn’t necessarily come from memorization.

"Repetition is the mother of learning, the father of action, which makes it the architect of accomplishment."

"Another strategy that works well in both humor and famous EDM drops is "violating the expectation", but you need to know the expectation before you can violate it. Once you learn music theory, you're able to violate it in ways that resonate."

You absolutely do not need to know anything about music theory to know how to play with expectation of a drop. If you aren't just feeling what the right thing is to do then you're not being creative, you're just following a set of predefined rules, which is likely to be dull.

For me creativity comes about when the brain stops 'actively thinking' - when I enter a flow state and creative things 'just happen' because of that zen like state. I get that when I DJ, play the guitar, play the piano, or even when I write code. It's like the subconscious brain has taken over for a while and is working on instinct.

It's true that you don't get to that state if you haven't practised enough. That's primarily because you don't want to be actively thinking about hitting the right key, finding the right fret, or knowing how something is architected. That active thinking will kill the creative flow, stone dead.

I'd argue creativity isn't memory in that sense, you need (memorised) technique, but it seems far too reductive to say that the creativity that emerges from ones subconscious is only memory -- it's clearly 'something else'.

I've also met plenty of smart people in my life who are creatively bereft!